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ABSTRACT 


Training  system  devices  are  frequently  used  for  aviation  training  to  prepare  students  to 
fly  aircraft.  The  use  of  training  systems  can  be  used  to  reduce  the  number  of  flight  hours 
required  for  pilots  and  aircrew.  The  aviation  training  system  device  must  be  designed 
properly  to  ensure  that  necessary  learning  objectives  are  met. 

Certification  is  the  last  step  in  the  test  and  evaluation  process  during  the 
validation  phase,  within  the  systems  engineering  process,  that  ensures  the  system  works 
as  it  was  intended,  and  meets  the  user’s  need.  Training  System  certification  ensures  the 
user  that  the  training  device  can  be  used  to  properly  meet  certain  learning  objectives  prior 
to  flying. 

This  thesis  analyzes  existing  training  system  device  certification  processes  and 
provides  recommendations  to  the  United  States  Navy,  Naval  Air  Warfare  Center  Training 
Systems  Division,  for  improvements. 
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EXECUTIVE  SUMMARY 


This  research  describes  and  analyzes  existing  certification  or  qualification  guidance  for 
aviation  training  system  devices.  Aviation  training  system  devices  are  used  to  help 
prepare  and  train  for  aircraft  flying.  The  rising  cost  of  fuel,  the  critical  skills  required  for 
flying,  and  the  cost  of  new  training  systems  devices  are  presenting  an  opportunity  for 
Naval  Air  Warfare  Center  Training  Systems  Division  (NAWCTSD)  to  examine  its 
certification  and  qualification  process  for  aviation  training  systems. 

To  ensure  the  systems  can  be  used  properly  to  meet  the  original  learning 
objectives  for  the  pilot  or  aircrew,  the  system  must  be  designed  and  tested  properly  to 
ensure  it  meets  the  user’s  goals  and  needs.  There  are  several  systems  engineering 
methods  that  can  be  used  to  ensure  the  final  training  system  device  meets  the  original 
design  intent.  Training  System  certification,  as  a  stage  in  the  systems  engineering 
process,  ensures  the  user  that  the  training  device  can  be  used  to  properly  meet  certain 
learning  objectives  prior  to  or  substitution  for  actual  flying. 

This  thesis  begins  with  a  top-level  data  analysis  of  the  different  guidance 
available  for  training  system  management  and  training  system  certification.  All  of  the 
military  organizations’  guidance  is  directly  linked  to  the  Department  of  Defense 
Instruction  5000.02  (DoDI  5000.02),  Operation  of  the  Defense  Acquisition  System.  The 
Federal  Aviation  Administration  (FAA)  qualification  process  is  traceable  back  to  the 
Title  14  Code  of  Federal  Regulations  (CFR)  requirements.  Ultimately,  all  the  training 
device  certification  or  qualification  guidance  is  traceable  back  to  one  overarching 
instruction.  This  thesis  explains  the  training  system  device  guidance  structure  for  each 
organization  for  certification  or  qualification,  as  well  as  provides  recommendations  to 
NAWCTSD  to  improve  its  own  guidance. 

Through  analysis  of  all  the  existing  training  system  device  certification  and 
qualification  processes,  this  thesis  provides  recommendations  to  include  the  necessary 
stakeholders  in  the  requirements  generation  phase  and  throughout  the  acquisition 
program,  to  include  a  certification  or  qualification  process  as  part  of  the  acquisition 
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program,  and  to  test  the  training  system  device  for  the  ability  to  meet  the  original  learning 
objectives  for  Training  and  Readiness  (T&R). 

Further  areas  of  study  are  required  to  include  unpublished  internal  guidance  from 
each  organization.  There  is  a  possibility  that  additional  guidance  exists  for  training 
system  certification  or  qualification  but  is  not  available  to  other  organizations.  Other 
potential  areas  of  research  would  include  providing  a  cost-benefit  analysis  for  offsetting 
aircraft  training  with  aviation  training  devices. 


xx 


ACKNOWLEDGMENTS 


I  thank  God  for  the  wonderful  opportunity  to  be  selected  by  NAVAIR  for  the 
SEM  PD21  program. 

I  wish  to  thank  my  wonderful  wife,  Andrea,  and  our  three  children,  Sarah,  Luke, 
and  Elizabeth,  for  their  encouragement  and  patience  over  the  past  two  years  of  our  lives.  I 
look  forward  to  having  fun  with  my  family  now  that  I  have  the  time  to  spend  the 
evenings  and  weekends  with  them  once  again. 

I  am  in  debt  to  Mr.  Joseph  Wascavage  for  encouraging  me  to  apply  to  this 
program,  and  Mr.  Randy  Geis,  Mr.  Robin  Locksley,  Mr.  Carl  Lee,  and  all  of  the 
leadership  at  NAWCTSD  for  giving  me  the  confidence  to  finish  the  program,  as  well  as 
the  time  to  work  on  my  studies. 

I  would  like  to  thank  Dr.  Rudolf  Darken  for  engaging  in  conversations  about 
training  system  devices  and  inspiring  me  to  conduct  a  thorough  analysis  on  existing 
training  system  device  certification  documentation  and  its  meaning.  I  also  would  like  to 
thank  Ms.  Mary  Vizzini,  Ms.  Barbara  Berlitz,  and  Ms.  Heather  Hahn  for  helping  me 
write  an  informative  thesis,  stay  on  track,  and  meet  the  thesis  deadlines  for  a  September 
graduation. 

Finally,  I  would  like  to  thank  Cohort  12  for  sticking  together  and  sharing 
information  and  encouragement  to  help  the  entire  class  leam  from  one  another’s 
successes  and  mistakes. 


xxi 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


I. 


INTRODUCTION 


A.  BACKGROUND 

Within  Naval  Air  Systems  Command  (NAVAIR),  the  Naval  Air  Warfare  Center 
Training  Systems  Division  (NAWCTSD)  is  the  Navy’s  source  for  a  full  range  of 
innovative  products  and  services  that  provide  complete  training  solutions.  This  includes 
requirements  analysis,  design,  development  and  full  life  cycle  support.  The  demands 
evolving  from  changes  to  modern  combat,  new  roles  for  military  operations,  and  the  use 
of  high  technology  weapons  systems  place  increased  emphasis  on  effective  and  efficient 
training  solutions. 

Aircraft  simulator  training  flight  hours  have  increased  in  recent  years  due  to  rising 
fuel  costs.  Training  systems  are  becoming  a  more  attractive  alternative  as  a  lower  cost 
training  option  to  provide  Training  and  Readiness  (T&R).  As  a  result,  greater  emphasis 
will  be  required  to  ensure  that  new  and  existing  training  systems  are  providing  the  proper 
skills  and  attributes  for  the  training  mission  and  is  not  degrading  the  training  skills.  To 
ensure  the  training  system  is  providing  the  necessary  potential  for  T&R  credit,  a 
certification  and  qualification  process  must  be  developed.  This  thesis  will  evaluate 
existing  training  device  certification  and  qualification  processes  for  aviation  and  examine 
the  requirements  from  the  different  agencies  that  generated  their  certification  and 
qualification  process.  Based  on  the  results  of  this  analysis,  the  thesis  will  provide 
recommendations  to  NAWCTSD  for  certification  or  qualification  of  their  aviation 
training  system  devices. 

B.  PURPOSE 

The  purpose  of  this  thesis  is  to  analyze  existing  training  system  devices 
certification  that  exists  for  other  organizations  and  determine  why  they  developed  the 
process.  This  analysis  will  be  used  to  provide  recommendations  to  NAWCTSD  for  its 
qualification  and  certification  process. 
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C.  RESEARCH  QUESTIONS 

•  What  were  the  stakeholder's  requirements  for  the  training  systems?  How 
are  the  requirements  specified,  and  how  are  they  subsequently  used  in  the 
development  process? 

•  What  organizations  currently  have  a  training  system  certification  and 
qualification  process?  Are  there  any  well  accepted  “best  practices”  in  the 
industry? 

•  Why  did  the  organization  develop  a  certification  and  qualification 
process? 

•  What  is  the  benefit  to  developing  a  certification  and  qualification  process? 
Can  it  be  quantified? 

•  What  is  an  acceptable  reporting  process  for  the  certification  process? 

•  What  are  the  tradeoffs  for  not  completing  all  of  the  recommended  test 
events  for  certification?  What  is  the  true  “return  on  investment”  made  in 
certification  and  qualification  process? 

D.  BENEFITS  OF  STUDY 

This  thesis  will  provide  NAWCTSD  a  recommendation  for  Naval  Aviation 
training  system  device  certification  and  qualification.  This  may  result  in  cost  reductions 
because  fewer  aircraft  flight  hours  are  used  for  training. 

E.  SCOPE 

This  thesis  focuses  on  providing  a  recommendation  to  NAWCTSD  by  analyzing 
existing  processes  for  training  system  devices.  The  analysis  will  be  dependent  on  existing 
documentation. 

F.  METHODOUOGY 

•  Conducted  literature  review  of  training  systems  documentation  for  training 
system  qualifications,  certifications,  requirements,  procedures, 
instructions. 

•  Analyzed  regulations,  policy,  procedures,  and  guidance  to  determine 
shortcomings. 

•  Developed  recommendations  for  improving  or  writing  guidance  for 
NAWCTSD  training  system  certification  and  qualification  process. 
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II.  TRAINING  DEVICE  CERTIFICATION  AND 
QUALIFICATION  PROCESS 

A.  INTRODUCTION 

In  order  to  provide  a  recommendation  for  training  system  certification  and 
qualification,  it  is  important  to  understand  what  an  aviation  training  system  device  is  and 
what  is  certification  or  qualification.  This  chapter  will  examine: 

•  What  is  an  aviation  training  system  device? 

•  When  does  certification  and  qualification  occur  within  the  systems 
engineering  process? 

•  What  is  the  purpose  of  certification  or  qualification  for  an  aviation  training 
system  device? 

There  several  different  types  of  aviation  training  systems  devices  as  well  as 
different  levels  of  simulation.  This  chapter  will  focus  on  reviewing  the  different  types  of 
aviation  training  devices  and  its  purpose.  After  understanding  what  an  aviation  training 
device  is  designed  for,  the  later  chapters  will  identify  the  importance  of  certification  or 
qualification  and  discuss  any  shortcomings  of  the  current  Federal  Aviation 
Administration  (FAA)  and  military  qualification  process. 

B.  AVIATION  TRAINING  SYSTEM  DEVICES 

Prior  to  World  War  II,  the  United  States  was  in  a  vulnerable  position  having 
thousands  of  individuals  without  a  military  background  who  had  to  be  trained  quickly  to 
operate  sophisticated  military  equipment.  The  new  military  pilots  had  to  be  trained  in  the 
ways  of  the  military  to  be  molded  into  combat-ready  crews  (Jenkinson  1983). 

To  accomplish  this  goal,  the  military  created  training  systems  rather  than  use 
military  equipment  and  to  avoid  costly  mistakes  during  training.  Military  training  today  is 
completely  different  than  it  was  before  World  War  II.  Military  pilot  training  consists  of 
classroom  training,  simulated  training  events,  and  flight  training  events.  The  classroom 
training  uses  a  combination  of  lectures  and  computer  based  modules  for  the  student  pilot 
to  learn  the  basics.  Flight  training  devices  are  available  in  different  configurations. 


3 


The  most  common  configuration  for  the  Navy  is  called  an  Operational  Flight 
Trainer  (OFT).  An  OFT  is  a  training  device  that  includes  the  pilot  and  navigator  or 
weapons  officer  operating  together  as  one  network.  A  simulator  tries  to  replicate  the 
configuration  of  the  aircraft  or  the  helicopter,  and  the  OFT  has  the  most  accurate 
configuration.  This  configuration  is  used  to  help  the  pilot,  navigator,  or  weapons  officer 
train  to  a  specific  mission  the  aircraft  is  designed  to  deliver.  Most  OFTs  have  full  motion 
that  replicates  the  flying  motion  of  the  real  aircraft.  The  movement  helps  provide  a 
realistic  environment  for  the  pilots,  navigator,  and  weapons  officers  as  they  perform  their 
mission  tasks.  Figure  1  is  what  a  MH-60  OFT  looks  like  from  the  exterior.  The  interior  is 
similar  to  a  MH-60R  cockpit. 


Figure  1.  MH-60R  Tactical  OFT.  (Photograph  by  Michael  C.  Barton.) 

A  Flight  Training  Device  (FTD)  is  similar  to  an  OFT.  A  FTD  lacks  motion  but 
has  a  full  visual  system  of  the  exterior  environment.  A  FTD  is  a  full  scale  replica  of  an 
airplane’s  instruments,  equipment,  panels,  and  controls.  A  FTD  can  be  configured  like 
the  airplane’s  cockpit  or  it  can  be  an  open  deck  configuration  without  the  small  confined 
area  under  a  canopy.  The  configuration  of  the  FTD  depends  on  how  the  instructors  plan 

to  use  the  training  device.  An  open  deck  area  is  better  for  beginners  so  the  instructor  can 
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provide  over  the  shoulder  mentoring  during  training  exercises.  An  enclosed  FTD  similar 
to  the  cockpit  is  often  used  for  pilots  to  maintain  efficiency.  The  FTD  does  not  require 
motion  or  visual  system  to  meet  the  criteria  outlined  by  the  learning  objective.  If  the 
learning  objective  requires  motion  cueing  a  different  configuration  training  device  would 
be  used  such  as  an  OFT.  There  is  no  set  configuration  requirement  for  an  aviation 
training  device  to  be  classified  as  an  FTD  (AC  120-45A  1992). 

The  FTD  shown  in  Figure  2  is  for  the  King  Air  350  Pro  Line  21  aircraft.  The 
system  does  not  have  the  motion  system  like  Figure  1  but  has  the  full  visual  system  and 
cockpit  like  the  aircraft  (Wood  2009). 


Figure  2.  King  Air  350  Pro  Line  21  FTD.  (Photograph  by  Ron  Csuy.) 

A  Part  Task  Trainer  (PTT)  is  a  training  device  that  does  not  have  a  motion  base 
like  an  OFT  or  full  visual  system  like  a  FTD.  Most  PTTs  are  used  for  a  specific  purpose 
such  as  instrument  familiarization.  The  PTT  helps  a  student  pilot  or  Naval  Flight  Officers 
(NFOs)  become  familiar  with  the  cockpit  or  other  missions.  This  type  of  training  device 
can  be  used  prior  to  students  entering  the  OFT  or  can  be  a  standalone  training  device.  As 
shown  in  Figure  3,  the  example  PTT  does  not  have  as  complex  of  a  visual  system  as  an 
FTD,  or  a  full  motion  system  as  an  OFT.  The  PTT  is  part  of  the  new  P-8A  Mission 
Operator  PTT  used  for  individual  refresher  training  (Pierce  2012). 
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Figure  3.  P-8A  Mission  Operator  PTT.  (Photograph  by  Clark  Pierce.) 


C.  THE  SYSTEMS  ENGINEERING  PROCESS 

The  certification  and  qualification  process  is  part  of  the  test  and  evaluation  (T&E) 
phase  in  the  systems  engineering  process.  Systems  engineering  has  been  defined  in  many 
different  ways  but  the  definitions  usually  have  the  same  goal.  The  International  Council 
on  Systems  Engineering  (INCOSE)  defines  of  systems  engineering  as: 

Systems  engineering  is  an  interdisciplinary  approach  and  means  to  enable 
the  realization  of  successful  systems.  It  focuses  on  defining  customer 
needs  and  required  functionality  early  in  the  development  cycle, 
documenting  requirements,  then  proceeding  with  design  synthesis  and 
system  validation  while  considering  the  complete  problem.  (INCOSE 
2004,  www.incose.org) 

The  Department  of  Defense,  Defense  Acquisition  Guidebook  (DAG)  defines  systems 
engineering  as: 

Systems  Engineering  (SE)  is  an  interdisciplinary  approach  encompassing 
the  entire  technical  effort  to  evolve  and  verify  an  integrated  and  total  life 
cycle  balanced  set  of  system,  people,  and  process  solutions  that  satisfy 
customer  needs.  SE  is  the  integrating  mechanism  across  the  technical 
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efforts  related  to  the  development,  manufacturing,  verification, 
deployment,  operations,  support,  disposal  of,  and  user  training  for  systems 
and  their  life  cycle  processes;  and  SE  develops  technical  information  to 
support  the  program  management  decision-making  process.  ( Defense 
Acquisition  Guidebook  2009,  4.1) 

The  systems  engineering  process  can  be  applied  to  any  system  that  is  being 
designed  to  accomplish  a  purpose.  For  this  thesis,  the  SE  process  is  used  for  pilot 
training  system  design  and  testing.  The  T&E  phase  of  the  SE  process  is  shown  in 
Figure  4.  The  T&E  process  is  the  right  side  of  the  “V”  and  tests  the  system  at  a 
component  level  up  to  the  system  level,  or  the  entire  training  system  for  this 
thesis.  This  thesis  will  focus  on  the  top  of  the  right  side  of  the  “V”  during  the 
validation  phase  where  qualification  and  certification  takes  place  at  a  training 
systems  level  just  prior  to  transition  to  the  fleet  or  customer  for  use. 


Figure  4.  Systems  Engineering  Process  (After  Defense  Acquisition  Guidebook  2009, 

4.1) 

D.  CERTIFICATION  AND  QUALIFICATION  PROCESS 

To  ensure  the  training  systems  are  an  accurate  representative  system  to  use  for 

training,  a  disciplined  approached  called  certification  or  qualification  must  be  used  to 

ensure  the  training  device  is  similar  to  and  closely  replicates  the  aircraft.  This  method  is 
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used  to  ensure  the  student  pilot  is  learning  on  a  system  that  is  close  to  the  aircraft.  The 
training  system  device  must  look  and  fly  like  the  aircraft.  If  it  does  not,  there  is  the 
potential  for  negative  training  to  occur  where  the  student  experiences  the  aircraft  not 
operating  like  he  or  she  thought  it  would,  based  on  having  used  the  training  device. 

Certification  is  the  last  step  in  the  T&E  process  during  the  validation  phase  that 
ensures  the  system  is  working  as  it  was  designed  to  and  meets  the  user’s  need. 
“Certification  is  a  formal  statement  by  the  architect  to  the  client,  or  user,  that  the  system, 
as  built,  meets  the  criteria  for  client  acceptance”  (Maier  and  Rechtin  2009,  17). 

Thus,  as  Maier  and  Rechtin  describe  in  this  step,  the  certification  process  can  be 
defined  as  a  formal  statement  to  the  training  system  customer  that  the  system,  as  built, 
meets  the  intent  of  the  training  system  goal.  In  other  words,  the  training  device  meets  the 
criteria  for  training  and  simulates  the  aircraft  device  similar  to  the  operational  scenario  it 
is  intended  to  model. 
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III.  TRAINING  SYSTEM  DEVICE  CERTIFICATION  AND 
QUALIFICATION  PROCESS  REVIEW 

A.  INTRODUCTION 

This  chapter  will  answer  the  following  research  questions: 

•  What  organizations  currently  have  a  training  system  certification  and 
qualification  process? 

•  Are  there  any  well  accepted  “best  practices”  in  the  industry? 

As  indicated  in  the  previous  chapter,  certification  and  qualification  is  the  last  step 
in  the  T&E  phase  of  the  SE  process  prior  to  delivering  the  training  system  to  the  fleet  or 
end  user.  The  certification  or  qualification  process  is  the  last  to  ensure  the  training  system 
device  is  working  correctly  and  the  device  is  fulfilling  the  requirements  for  the  user.  If 
this  is  not  followed  correctly,  the  fleet  or  user  will  determine  if  the  training  system  device 
is  working  properly  or  is  fulfilling  its  pilot  training  need. 

This  thesis  researched  several  different  military,  commercial,  and  federal  agencies 
to  develop  a  list  of  existing  certification  and  qualification  processes.  The  next  sections  in 
this  chapter  will  describe  the  existing  certification  and  qualification  guidance  for  training 
systems. 

B.  FEDERAL  AVIATION  ADMINISTRATION  PROCESS 

The  FAA  regulations  that  govern  aircraft  are  found  in  Title  14  of  the  Code  of 
Federal  Regulations  (CFR).  There  are  68  regulations  organized  into  three  volumes  under 
Title  14,  Aeronautics  and  Space.  The  forth  volume  is  for  the  Department  of 
Transportation  (DOT),  and  the  fifth  volume  is  for  the  National  Aeronautics  and  Space 
Administration  (NASA).  Figure  5  is  a  depiction  of  how  Title  14  is  organized.  (Aviation 
Technician  Handbook  n.d.) 
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Title  49  Transportation 


I 


Congressional  Acts 

lActs  of  Congress  are  Public  Law) 

1 


Code  of  Federal  Regulations  (CFR) 

<50  Titles) 

Public  Law  Bash  for 
Title  14  is  PL  103-272 

~  -"i 

Title  14  Aeronautics  and  Space 

(Five  Volumes) 


Volume  1 

Volume  2 

Volume  3 

Parts  1-59, 

Parts  60-139. 

Parts  140-199, 

Chapter  1 

Chapter  1 

Chapter  1 

Volume  4 
Chapter  II 
Parts  200 -399 
Office  of  Secretary, 
D.O-T. 

Chapter  III 
Parts  200 -399 
Parts  400-1199 
Office  of  Commercial 
Space  Transportation 


Volume  5 
Parts  1200-1299 
Chapter  V 


NASA 


Subchapter  A 

Definitions  and  Abbreviations 
Part:  1 , 3 

Subchapter  B 

Procedural  Rules 
Parts:11,13, 14, 15, 16, 17 

Subchapter 

Aircraft 

Parts:  21, 23, 2S,  27, 29, 31. 33, 
34.35.36.39.43,45.47,49 


Subchapter  D 

Airmen 

Parts:  61,63,65,67 

Subchapter  E 

Airspace 
Parts:  71. 73, 77 

Subchapter  F 

Air  Traffic  and  General 
Operating  Rules 
Parts:  91,93,95.97.99,101. 
103,105 

Subchapter  G 

Air  Carriers 

Parts:  119, 121, 125,129, 133, 
1 35, 1 36, 1 37, 1 39 


Subchapter  H 

Schools  and  Other  Certified 
Agencies 

Parts:  141, 142, 145, 147 

Subchapter  I 

Airports 

Parts:  1 50. 151.1 52, 1 55. 1 56, 1 57. 
158,161,169 

Subchapter  J 

Navigational  Facilities 
Parts:  170, 171 

Subchapter  K 

Administrative  Regulations 
Parts:  183, 185,1 87, 189, 193 

Subchapter  L  through  M 

Reserved 

Subchapter  N 

Risk  Insurance 
Part:  198 


Figure  5.  CFR  Title  14  Structure  (From  Aviation  Technician  Handbook  n.d.,  12-2) 


The  FAA  rules  are  referred  to  as  Federal  Aviation  Regulations  (FARs).  The  FAA 
FAR  is  often  confused  with  another  set  of  government  regulations  called  the  Federal 
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Acquisition  Regulation,  also  called  FAR,  so  the  FAA  regulations  are  referred  to  as  “Title 
14  CFR.”  (Federal  Aviation  Administration  n.d.)  Table  1  lists  all  the  Volumes,  Chapters, 
and  Parts  for  Title  14  CFR.  The  various  parts  of  Title  14  listed  in  this  table  are  available 
electronically  on  the  U.S.  Government  Printing  Office  (GPO)  website.  Part  60  of  the 
CFR,  covers  the  flight  simulation  training  device  initial  and  continuing  qualification. 
(Federal  Aviation  Administration  2013) 


Table  1.  Title  14  CFR  Parts  (From  Federal  Aviation  Administration  2013, 

www.ecfr.gov) 


Title 

Volume 

Chapter 

Browse  Part 

Regulatory  Entity 

Title  14 
Aeronautics 
and  Space 

1 

1 

1-59 

Federal  Aviation  Administration, 

Department  of  Transportation 

2 

60-109 

3 

110-199 

4 

II 

200-399 

Office  of  the  Secretary, 

Department  of  Transportation 
(Aviation  Proceedings) 

III 

400-1199 

Commercial  Space  Transportation. 

Federal  Aviation  Administration, 

Department  of  Transportation 

5 

V 

1200-1299 

National  Aeronautics  and  Space  Administration 

VI 

1300-1399 

Air  Transportation  System  Stabilization 

The  FAA  has  several  Advisory  Circulars  (ACs)  that  provide  additional  guidance 
to  assist  the  aviation  community  to  comply  with  Title  14  CFR.  In  the  particular,  the  ACs 
provides  guidance  for  everything  related  to  complying  with  Title  14  CFR  for  aviation. 
There  are  four  ACs  available  for  the  qualification  and  certification  of  aircraft  simulators 
used  in  training  programs  or  for  airmen.  The  four  ACs  provide  guidance  to  comply  with 
Title  14  CFR,  Part  60. 

There  is  an  AC  for  each  of  the  different  aircraft  simulator  types.  For  example, 
Airplane  Simulator  Qualification ,  AC  120-40B,  covers  all  aircraft  qualification 
requirements  to  comply  with  Title  14  CFR.  Table  2  lists  the  ACs  for  all  the  different 
aircraft  training  system  devices  from  fixed  wing  to  rotary  wing.  (Federal  Aviation 
Administration  (FAA)  National  Simulator  Program  (NSP)  n.d.) 
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Table  2.  FAA  ACs  for  Fixed  Wing  and  Rotary  Wing  Simulators 


Advisory  Circular 

Subject 

Date 

120-40B 

Airplane  Simulator  Qualification 

7/29/1991 

120-45A 

Airplane  Flight  Training  Device 
Qualification 

2/5/1992 

120-63 

Helicopter  Simulator  Qualification 

10/11/1994 

121-14C 

Aircraft  Simulator  and  Visual  System 
Evaluation  and  Approval 

8/29/1980 

C.  UNITED  STATES  AIR  FORCE  PROCESS 

The  United  States  Air  Force  (USAF)  training  system  device  certification  and 
qualification  process  is  outlined  in  Air  Force  Instruction  36-2251  (AFI  36-2251), 
Management  of  Air  Force  Training  Systems  (2009).  Similar  to  the  FAA,  the  AFI  is 
traceable  back  to  one  overarching  regulation,  the  Department  of  Defense  Directive 
5000.01  (DoDD  5000.01)  and  Department  of  Defense  Instruction  5000.02  (DoDI 
5000.02).  DoDI  5000.02  provides  guidance  for  engineering,  acquisition,  and  testing  for 
new  and  modified  DoD  systems.  The  DAG  provides  further  clarification  of  the 
requirements  listed  in  the  DoDI  5000.02  (AFI  36-2251  2009). 

AFI  36-2251  provides  guidance  for  managing  USAF  training  systems.  “It 
outlines  the  requirement  to  develop,  acquire,  modify,  test,  validate,  and  support  training 
systems,  to  include  but  not  limited  to  Aircrew  Mission  Training  Systems,  Mission  Crew 
(i.e.  Command  and  Control  (C2))  Training  Systems,  Maintenance  Training  Systems, 
Space  Training  Systems,  other  Training  Systems  and  Training  Services”  (AFI  36-2251 
2009,  3).  AFI  36-2251  is  used  in  conjunction  with  Air  Force  Instruction  10-601  (AFI 
10-601),  Operational  Capability  Requirements  Development  (2010),  Air  Force 
Instruction  99-103  (AFI  99-103),  Capabilities-Based  Test  and  Evaluation  (2009),  and 
Air  Force  Instruction  (AFI  63-101),  Integrated  Life  Cycle  Management  (2013).  AFI  36- 
225 1  provides  an  integrated  framework  for  the  implementation  of  a  training  system  (AFI 
36-2251  2009). 

This  thesis  will  not  analyze  the  DoDI  5000  series  but  will  review  the  AFIs  related 
to  training  systems  certification  and  qualification.  USAF  has  another  instruction  that  also 
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covers  what  is  referred  to  as  Simulator  Certification  (SIMCERT)  in  Air  Force  Instruction 
36-2248  (AFI  36-2248),  Operation  and  Management  of  Aircrew  Training  Devices 
(1998).  There  does  not  appear  to  be  a  connection  between  AFI  36-2248  and  AFI  36- 
2251.  Both  instructions  cover  the  same  topic  of  aviation  training  device  certification  and 
qualification  but  do  not  refer  to  either  instruction  for  additional  guidance.  The  scope  of 
AFI  36-2251  is: 

This  instruction  specifies  the  responsibilities  of  the  Combat  Air  Forces 
(CAF)  to  operate  and  manage  Training  Systems  (TSs),  including  Aircrew 
Training  Devices  (ATDs),  Training  System  Support  Centers  (TSSCs), 
associated  support  equipment,  courseware,  and  instruction.  The  CAF 
includes  the  following  agencies:  Headquarters  Air  Combat  Command  (HQ 
ACC),  Headquarters  United  States  Air  Forces  in  Europe  (HQ  USAFE), 
Headquarters  Pacific  Air  Forces  (HQ  PACAF),  Air  National  Guard 
(ANG),  Headquarters  Air  Force  Reserve  Command  (HQ  AFRC),  and 
Headquarters  Air  Education  and  Training  Command  (HQ  AETC).  (AFI 
36-2248  1998,  4) 

Figure  6  provides  a  visual  depiction  of  the  AFIs  and  DoDI  5000.02  relationship. 
This  relationship  is  similar  to  the  FAA  Title  14  CFR  overarching  requirement.  Unlike  the 
FAA,  the  AFIs  are  not  based  on  the  aviation  training  systems  device  type.  The  AFIs  are 
organized  to  comply  with  DoDI  5000.02  and  Air  Force  Policy  Directive  36-26  (AFPD 
36-26)  Total  Force  Development  (2011)  that  replaced  Air  Force  Policy  Directive  36-22 
(AFPD  36-22)  Military  Training  (2004).  AFI  36-2248  and  AFI  36-2251  cover 
SIMCERT. 
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Used  in  Conjunction  With: 


Figure  6.  USAF  Management  of  Training  Systems  AFI  Structural  Relationship 
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D.  UNITED  STATES  ARMY  PROCESS 


The  United  States  Army  (USA)  has  the  same  requirement  as  the  USAF  to  follow 
the  DoDD  5000.01  and  DoDI  5000.02.  After  a  considerable  amount  of  literature  research, 
it  is  a  safe  assumption  to  conclude  that  there  is  no  USA  specific  policy  or  guidance  for 
training  system  certification  or  qualification.  It  is  possible  that  that  the  USA  is  using 
existing  guidance  such  as  the  USAF  or  FAA  for  aviation  training  system  certification  and 
qualification.  Similar  to  the  USAF  AFIs,  the  Army  Regulations  (ARs)  are  arranged  to 
comply  with  the  overarching  DoDI  5000.02  requirements.  Aviation  training  device 
management  is  covered  under  Army  Regulation  350-38  (AR  350-38),  Policies  and 
Management  for  T mining  Aids,  Devices,  Simulators,  and  Simulations  (2013).  (AR  350- 
38  2013) 

AR  350-38  “establishes  Army  policies  and  responsibilities  for  life  cycle 
management  of  the  following  areas  only  as  they  pertain  to  training:  training  aids,  devices, 
simulators,  and  simulations  (TADSS),  including  tactical  engagement  simulation  (TES), 
targetry,  combat  training  centers,  gaming  technologies,  range  instrumentation,  and 
training- unique  ammunition,  regardless  of  training  site  or  event  (combat  training  centers, 
homestations,  institutions,  or  other  training  sites  or  venues)”  (AR  350-38  2013,  1).  This 
regulation  also  expands  upon  Army  Embedded  Training  (ET)  as  stated  in  Army 
Regulation  350-1  (AR  350-1),  Army  Training  and  Leader  Development  (2011).  ET  is  a 
subset  of  Army  training  for  systems  training.  Both  AR  350-38  and  AR  350-1  are 
provided  to  training  systems  managers  as  guidance  to  field  training  systems  to  the  user 
quickly  and  efficiently.  However,  these  regulations  do  not  mention  a  requirement  for 
training  system  certification  or  qualification. 

Figure  7  provides  a  visual  depiction  of  AR  350-38  and  DoDI  5000.02 
relationship.  This  relationship  is  similar  to  the  USAF  but  AR  350-38  does  not  provide  a 
certification  or  qualification  process  for  aviation  training  devices  like  AFI  36-2248  and 
AFI  36-2251  do  for  USAF  aviation  training  devices. 
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AR  350-38  Used  in  Conjunction  With: 


Figure  7.  USA  Training  Device  Management  AR  Structural  Relationship 
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E.  UNITED  STATES  NAVY  PROCESS 


The  USN  follows  a  similar  structure  to  the  USAF  and  USA  but  does  not  have  a 
specific  instruction  for  aviation  training  system  management,  certification  or 
qualification.  The  Department  of  the  Navy  (DON)  does,  however,  have  two  different 
processes  called  Training  Device  Certification  and  Accreditation  Process  (TDCAP)  and 
Systematic  Team  Assessment  of  Readiness  (START)  (Owen  and  Meyers  2012). 

The  START  and  TDCAP  process  leverage  off  of  existing  DoD,  Secretary  of  the 
Navy  (SECNAV),  and  NAVAIR  instructions  to  provide  certification  or  qualification  for 
aviation  training  devices.  All  the  Navy  instructions  are  traced  back  to  the  DoDD  5000.01 
and  DoDI  5000.02.  The  Navy  Instructions  are  slightly  different  than  the  USAF  and  USA. 
The  Navy  has  an  additional  layer  of  instructions  within  NAVAIR  to  provide  clarification 
for  DoD,  SECNAV,  and  Chief  of  Naval  Operations  (OPNAV)  instructions.  Figure  8 
shows  the  available  instructions  and  Navy  processes  for  aviation  training  systems 
certification  or  qualification  and  their  relationship  to  DoDI  5000.02. 
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Used  in  Conjunction  With: 


Figure  8.  Available  USN  Training  Systems  Certification  or  Qualification  Instructions  or  Guidance 
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IV.  ANALYSIS  OF  THE  EXISTING  TRAINING  SYSTEM  DEVICE 
CERTIFICATION  AND  QUALIFICATION  GUIDANCE 

A.  INTRODUCTION 

This  chapter  will  answer  the  remaining  research  questions: 

•  What  were  the  stakeholder's  requirements  for  the  training  systems?  How 
are  the  requirements  specified,  and  how  are  they  subsequently  used  in  the 
development  process? 

•  Why  did  the  organization  develop  a  certification  and  qualification 
process? 

•  What  is  the  benefit  to  developing  a  certification  and  qualification  process? 
Can  it  be  quantified? 

•  What  is  an  acceptable  reporting  process  for  the  certification  process? 

•  What  are  the  tradeoffs  for  not  completing  all  of  the  recommended  test 
events  for  certification?  What  is  the  true  “return  on  investment”  made  in 
certification  and  qualification  process? 

As  seen  in  the  previous  chapter,  the  FAA  certification  and  qualification  process 
flows  down  from  the  Title  14  CFR  to  the  corresponding  AC  that  provides  guidance  for 
complying  with  Title  14  CFR.  The  various  ACs  are  arranged  according  to  aircraft  type. 
For  the  USAF,  USA,  and  USN,  the  requirements  are  based  on  acquisition  regulation  and 
they  all  flow  down  from  the  DoDD  5000.01.  The  certification  and  qualification  process 
are  included  within  the  USAF,  USA,  and  USN  instructions  or  regulations. 

This  chapter  presents  how  the  existing  instructions  address  the  certification  and 
qualification  process  for  aviation  training  systems.  It  will  provide  the  USN  stakeholder’s 
requirements  for  aviation  training  systems  certification  or  qualification  to  determine 
whether  the  existing  instructions  meet  or  lack  the  guidance  required  for  meeting  them. 

B.  STAKEHOLDER  REQUIREMENTS 

The  DON  uses  a  combination  of  aviation  training  devices  and  aircraft  to  provide 
T&R.  Owen  and  Meyers  provide  a  good  summary  of  the  USN  and  United  States  Marine 
Corps  (USMC)  stakeholder  requirements.  “Given  the  constrained  fiscal  environment  now 
and  in  the  foreseeable  future,  the  use  of  aircraft  flight  hours  for  training  and  skill 
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qualification  is  a  costly  solution  to  maintain.  Thus,  the  use  of  simulation  is  becoming  an 
even  more  attractive  alternative  to  aircraft  training  flight  hours.”  (Owen  and  Meyers 
2012,  1).  As  a  result,  USN  and  USMC  are  interested  in  looking  at  using  more  aviation 
training  system  devices  to  provide  more  T&R  credit  than  the  aircraft.  The  cost  to  use  a 
training  device  is  less  expensive  than  using  an  aircraft  for  training.  The  price  of  jet  fuel  is 
3.5  times  higher  in  2012  than  it  was  in  2000  (Airlines  for  America  2013).  A  training 
lesson  that  takes  two  hours  in  an  aircraft  costs  on  average  $3,500.  This  does  not  include 
maintenance  cost.  Maintenance  cost  would  be  an  additional  cost.  The  operating  cost  for 
an  aircraft  simulator  is  significantly  cheaper  since  it  only  uses  electricity.  The  same 
training  lesson  in  an  aircraft  simulator  would  cost  under  $100.  This  does  not  include 
maintenance  cost. 

A  certification  or  qualification  process  is  required  to  show  evidence  to  the  user 
that  the  training  system  device  can  be  used  for  T&R.  The  existing  USN  and  USMC 
acquisition  and  SE  process  does  not  cover  certification  or  qualification  for  new  aviation 
training  system  devices  to  determine  if  the  new  systems  are  meeting  T&R  requirements. 
The  current  acquisition  process  focuses  more  on  Statement  of  Work  (SOW)  and 
performance  specification  requirements  to  determine  if  the  training  system  device  was 
built  according  to  the  contract  and  that  it  was  built  correctly.  A  certification  or 
qualification  of  T&R  report  must  be  presented  to  the  end  user  to  document  the  capability 
of  the  new  training  device.  The  certification  or  qualification  report  must  show  the 
supporting  evidence  for  meeting  T&R.  This  process  determines  if  the  training  device  is 
the  right  system  for  providing  T&R,  not  just  for  ensuring  that  the  acquisition  contractual 
requirements  have  been  met  (Owen  and  Meyers  2012). 

The  USN  and  USMC  training  systems  require  an  engineering  process  to  ensure 
the  system  is  designed  and  tested  properly.  This  provides  evidence  that  the  training 
system  meets  the  original  learning  objectives  listed  in  the  T&R  matrix  for  that  skill  set. 
The  first  step  in  any  new  system  design  is  to  identify  the  stakeholders.  For  the  purposes 
of  this  thesis,  the  stakeholder  requirement  will  be  limited  to  the  USN  and  USMC.  The 
goal  of  the  aviation  training  system  is  to  ensure  that  the  learning  objectives  can  be 
accomplished  with  the  new  aviation  training  device.  This  will  serve  as  the  primary  goal 
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of  the  training  system.  The  form  should  follow  the  function  for  the  training  system  and 
ultimately  be  certified  or  qualified  back  to  the  original  learning  objectives  (Owen  and 
Meyers  2012).  This  method  is  not  any  different  from  designing  a  system  in  the  classical 
“form-follows-function”  systems  engineering  concept  (Maier  and  Rechtin  2009,  10). 
Owen  and  Meyers  show  the  common  stakeholders  for  USN  and  USMC  aviation  training 
system  devices. 


Fleet 

. 

TDCAP  Accred  tation  Authority 

TYCOM 

Establishes  T&R  standards 
Monitorstraining  readiness 
throughoutthe  training  cycle 


Aircraft  Model  Manager 

DefinesT&R  Credits  for  TD 
Assigns  FPT  aircrew 


Fleet  Cert  Team 

Cert/Accreditation  plan 
Participates  in  cert  and  recert 
Provide  fleets  inputs  on  TSR 
mappings  and  ratings 


CNAF  /  OPNAV  /  N88 

Sets  certification  policy  (via  3710.7U) 
Owns  Training  Device  Certificates 
Sets  T&R  Goals 
Funds  cert  and  re-cert 
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AIR-4. OP 

Cert  process  enforcer 
Maintain  master  cert  database 
Maintains  TVC  Configuration 
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Manages  Training  Device 
Technical  and  Certified  Baseline 
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AIR-4.6 

Cert/Accreditation  plan 
Participates  in  cert  and  recert 
Provides  engineering  support 
Trains  ISEO  for  recerts 


TDCAP  stakeholders  responsible  for  certifying  training  device 
■  _  _  Test  Team  tedby.  AIR5.1&4.6  ] . 


Flow  of  authority  /  requirements 
Flow  of  information 


Certs/Recerts  can  be  funded  via  PM  A  205  or 
directly  by  Platform  PMs  or  Program  Sponsors 


AIR-5.1 

Cert/Accreditation  plan 
Participates  in  cert  and  recert 
Provides  test  support 
Ensures  ISEO  and  Fleet  aircrew  adhere 
to  and  are  trained  on  T&E  processes 


. . 


Figure  9.  Stakeholders  for  USN  and  USMC  Aviation  Training  System  Devices.  (From 

Owen  and  Meyers  2012,  3) 


Section  C  will  analyze  the  existing  USN  guidance  and  identify  where  the  current 
guidance  lack  the  necessary  details  to  fulfill  the  USN  stakeholder’s  requirements.  Finally, 
it  will  analyze  existing  FAA,  USAF,  and  USA  instructions  and  regulations  to  determine 
if  the  guidance  meets  the  USN  certification  requirements. 
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C.  REVIEW  OF  EXISTING  USN  AVIATION  TRAINING  SYSTEM  DEVICE 

GUIDANCE 

The  NAWCTSD  acquisition  programs  follow  the  requirements  in  the  DoDI 
5000.02  and  NAVAIRINST  4355. 19D  for  systems  engineering.  NAVAIRINST 
4355. 19D  requires  that  each  training  system  acquisition  program  follow  the  Systems 
Engineering  Technical  Review  (SETR)  process.  Each  system  must  go  through  a  Systems 
Requirement  Review  I  (SRR-I)  to  determine  if  the  government  has  the  correct 
requirements  to  meet  the  goals  for  that  system.  NAWCTSD  reviews  the  requirements  to 
ensure  they  are  capable  of  meeting  the  learning  objectives  for  T&R  during  an  SRR-I. 

1.  Acquisition  Guidance  and  Instructions 

Aviation  training  system  acquisitions  verify  requirements  using  the  DoDI  5000.02 
process.  The  training  system  acquisition  team  uses  the  Front  End  Analysis  (FEA)  and 
training  system  Concept  of  Operations  (CONOPS)  to  develop  the  performance  or  system 
specification  according  the  Systems  Engineering  Plan  (SEP)  and  Military  Specification 
Standard  961E  (MIL-STD-961E).  This  process  requires  the  training  system  acquisition 
team  to  create  a  Requirements  Tractability  Verification  Matrix  (RTVM).  The  existing 
DoD,  SECNAV,  and  NAVAIR  instructions  allow  the  test  team  to  verify  that  the  training 
system  is  built  correctly.  Often,  the  training  system  is  not  tested  in  a  way  to  determine  if 
the  system  is  meeting  the  original  training  systems  goal  or  learning  objectives. 

2.  START  Process 

The  FEA  and  CONOPS  for  training  systems  are  not  available,  or  exist  in  a  draft 
state,  when  the  systems  engineering  process  starts  for  new  aviation  training  systems. 
Often,  the  original  requirements  for  the  aviation  training  system  change  during  the 
development  process  which  impact  the  training  system's  ability  to  meet  the  new  learning 
objectives.  The  START  process  was  developed  to  correct  the  current  issues  with  the 
current  acquisition,  systems  engineering,  and  test  process.  All  of  the  learning  objectives 
are  mapped  to  other  learning  objectives  and  T&R  events.  Owen  and  Meyers  list  the  six 
steps  in  the  START  process: 

1.  Tasks  are  decomposed  to  their  lowest  level 
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2.  Tasks  are  mapped  to  skills  by  criticality 

3.  Tasks  are  mapped  to  simulator  attributes  by  criticality  and  simulator  capability 

4.  A  gap  analysis  is  conducted  on  required  simulator  attributes  and  a  baseline  is 

set 

5.  Enhancements  and  impact  on  capabilities  are  identified 

6.  A  cost  benefit  analysis  on  candidate  upgrades  is  conducted  (Owen  and  Meyers 

2012,3) 

This  process  assesses  existing  simulators  to  determine  if  the  aviation  training 
device  meets  the  learning  objectives  and  T&R.  The  START  process  does  not  incorporate 
the  required  feedback  in  the  design  process  or  provide  guidance  for  certification  or 
qualification  for  aviation  training  devices. 

3.  Naval  Air  Training  and  Operating  Procedures  Standardization 

Instruction 

Naval  Air  Training  and  Operating  Procedures  Standardization  (NATOPS) 
instruction  is  OPNAVINST  3710.7U.  OPNAVINST  3710.7U  provides  guidance  for 
certifying  naval  pilots,  NFOs,  and  aircrew  training  for  logging  flight  time.  Aviation 
training  devices  can  be  used  as  an  acceptable  method  to  certify  flight  time  for  naval  pilots 
if  they  are  listed  in  Appendix  K  of  that  instruction.  The  instruction  does  not  describe  the 
process  to  add  or  remove  aviation  training  devices.  The  instruction  only  mentions  that 
“change  recommendations  to  approved  simulators  may  be  made  by  letter  to  Commander, 
Naval  Air  Forces  N455  (COMNAVAIRFOR  455).”  (OPNAVINST  3710.7U  2009,  K-l) 

According  to  Owen  and  Meyers,  the  lack  of  detail  in  Appendix  K  of 
OPNAVINST  3710.7U  provides  the  potential  for  misinterpretation  and  inconsistency  to 
the  method  of  adding  an  aviation  training  system  device  to  the  list  for  logging  training 
flight  time  (Owen  and  Meyers  2012). 

4.  T&E  Instructions 

Currently,  the  aviation  training  systems  follow  DoD,  SECNAV,  and  NAVAIR 
instructions.  The  DoDI  5000.02  uses  the  system  engineering  “V”  model  for  design  and 
verification.  SECNAV  5000. 2E  and  NAVAIRINST  3960. 2D  provide  guidance  for  naval 
acquisition  systems  to  comply  with  this  method.  The  existing  T&E  instructions  do  not 
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specifically  address  training  systems  certification  or  qualification  process  for  the  final 
stage  of  the  T&E  phase,  also  described  as  the  top  right  portion  of  the  systems  engineering 
“V”  model.  The  guidance  given  in  the  instructions  requires  the  T&E  strategy  to  test  the 
device  to  ensure  it  meets  the  original  system  requirements  or  goal.  The  guidance  does  not 
provide  the  necessary  details  to  confirm  that  the  training  device  meets  the  original 
learning  objectives  or  is  capable  of  meeting  T&R  credit.  The  guidance  provides  the 
foundation  for  the  certification  or  qualification  if  the  certification  process  is  defined  as  a 
requirement  early  in  the  acquisition  program.  It  is  easily  overlooked  if  the  design  team  is 
not  experienced  enough  to  understand  that  it  was  overlooked  and  not  identified  as  a 
stakeholder  requirement.  The  system  engineering  and  T&E  process  would  be  required  to 
derive  a  certification  or  qualification  process  to  meet  the  learning  objective  and  T&R 
requirement.  All  of  these  instructions  lack  specific  system  details  so  that  they  can  be 
applied  to  any  naval  acquisition  system. 

5.  T&R  Instructions  and  Guidance 

The  USN  uses  Training  Data  Products  Military  Performance  Specification 
29612B  (MIL-PRF-29612B)  and  Military  Handbook  29612-2A  (MIL-HDBK-26912- 
2A)  as  a  guide  for  the  Instructional  Systems  Development  (ISD)  group  at  NAWCTSD  to 
develop  instructional  materials  for  aviation  training  system  devices.  MIL-HDBK-26912- 
2A  does  not  provide  details  about  certification  or  qualification,  but  it  does  say  the  system 
must  be  evaluated.  “Evaluation  is  a  continuous  process  that  starts  during  the  analysis 
phase  and  continues  throughout  the  development  and  life  cycle  of  the  instructional 
system.  Feedback  from  the  evaluation  process  is  used  to  modify  the  training  program  as 
necessary”  (MIL-HDBK-29612-2A  2001,  8).  Feedback  is  required  for  fielded  training 
systems  from  both  internal  and  external  users.  Periodic  evaluations  are  critical  to  ensure 
the  training  system  meets  the  original  training  goal.  The  users  might  be  spending  more 
time  than  required  to  complete  the  necessary  training.  The  evaluation  may  suggest  a 
modification  to  correct  this  problem.  MIL-HDBK-29612-2A  suggests  using  FAA 
training  system  device  certification  or  qualification  when  required  but  does  not  direct  the 
developing  agency  or  engineering  team  to  use  it  exclusively  (MIL-HDBK-29612-2A 

2001).  Similar  to  OPNAVINST  3710.7U,  this  handbook  does  not  provide  enough  detail 

24 


to  evaluate  the  aviation  training  device.  The  handbook  is  designed  to  cover  all  learning 
objectives  and  T&R  for  all  the  different  types  of  instructional  systems,  not  just  aviation 
training  devices. 

The  USMC  uses  Navy  Marine  Corps  3500. 14C  (NAVMC  3500. 14C)  Aviation 
Training  and  Readiness  Program  Manual  (2011),  as  a  reference  to  develop  T&R  for 
USMC  aviation  programs.  NAVMC  3500. 14C  requires  a  committee  to  complete  a 
Training  Device  Event  Essential  Subsystem  Matrix  (EESM)  for  each  aviation  community 
that  is  using  a  training  device  for  T&R  credit.  Table  3  is  an  example  subsystem  list 
required  to  take  T&R  event  credit  in  the  aviation  training  device  (NAVMC  3500. 14C 
2011). 
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Table  3.  Example  Subsystems  Required  for  T&R  Credit  (After  NAVMC  3500. 14C 

2011,6-38) 


Aerodynamic  Model 

After  Action  Review  (Debrief  Station/  Debrief  Playback,  etc) 

Aircraft  Survivability  Equipment 

Aural 

Automatic  Flight  Control  System 

Caution  Warning  System 

Cockpit  Displays 

Cockpit  Instruments 

Cockpit  Panels 

Comnis/ICS 

Copilot/ Aircrew  Systems  (as  applicable) 

Flight  Controls 

Instructor  Operator  Station 

Landing  Gear  System  (as  applicable) 

lighting  System 

Mechanical  Diagnostic  System 

Miscellaneous  Mission  Systems 

Miscellaneous  Switches/knobs  (blade  fold,  anti-ice) 

Mission  Planning  Interface 

Motion  Systems 

Moving  Models  (Ships,  Aircraft,  Vehicles,  and  associated  capabilities/signatures) 

Navigation  systems 

Operational  Flight  Program/SCS  (current  flight  software) 

Sensor  Systems 

TEN/Networking  Capability 

Visual  System  (Environmental  Conditions,  Database  Coverage,  etc.) 

Visual  System 

Weapons  Systems 

NAVMC  3500. 14C  lists  example  subsystems  to  take  T&R  event  credit  but  it  does 
not  describe  how  these  forms  or  devices  meet  the  functional  requirements  for  T&R  event 
credit.  The  instruction  does  not  provide  the  proper  guidance  to  perform  a  proper  system 
decomposition  to  map  the  various  subsystems  or  form  to  the  T&R  event  credit  of 
functional  requirement.  This  instruction  lacks  the  guidance  to  perform  this  task  to  ensure 
the  aviation  training  system  device  will  be  designed  correctly  to  meet  the  T&R  event  or 
learning  objective  requirements. 
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6. 


TDCAP 


TDCAP  is  a  task-based/attribute  evaluation  to  determine  if  the  training  system  has 
the  ability  to  provide  the  required  training  by  examining  the  available  sensory  inputs  or 
attributes.  Example  attributes  examined  during  the  TDCAP  process  are  visual,  audio, 
touch  cues,  and  motion.  Each  attribute  is  analyzed  against  its  required  task  such  as 
execute  ground  taxi,  perform  air  intercept,  and  landings  for  example.  The  design  and  test 
team  need  to  understand  what  attributes  are  required  to  provide  proper  training. 

TDCAP  defines  the  training  device’s  attributes  required  to  support  the 
design,  development,  and  test  of  the  training  solution  to  meet  the  tasks 
associated  with  the  platform’s  T&R  events  and  LOs.  Using  both 
quantitative  and  qualitative  measures,  TDCAP  can  evaluate  the  fidelity  of 
the  training  device  against  the  platform’s  hardware  and  software 
configurations  including  flying  qualities,  air  vehicle  systems,  mission 
environments,  weapon  systems  capabilities,  and  distributed  training  to 
validate  the  capabilities  of  the  training  device  to  successfully  meet  the 
T&R  requirements  of  the  modeled  system.  (Owen  and  Meyers  2012,  4) 

As  Owen  and  Meyers  describe,  TDCAP  is  a  well-documented  process  that  meets 

all  the  DoD,  SECNAV,  and  NAVAIR  acquisition  instruction  requirements  discussed 

earlier  in  this  chapter.  TDCAP’ s  primary  goal  is  to  minimize  an  acquisition  program’s 

cost  and  schedule.  This  process  also  provides  additional  guidance  to  ensure  the 

requirement  generation  and  validation  processes  provide  an  objective  assessment  to 

determine  if  the  aviation  training  system  device  is  capable  of  meeting  T&R  events  or 

learning  objectives  (Owen  and  Meyers  2012). 

TDCAP  is  a  combination  of  MIL-HDBK-29612-2A  and  the  START  process.  The 
START  process  was  developed  to  examine  existing  training  devices  to  provide  an 
assessment  to  see  if  the  training  device  was  acceptable  for  taking  T&R  event  credit.  As  a 
result,  the  START  process  provides  evidence  that  the  training  device  is  capable  of 
meeting  T&R  event  credit,  or  the  process  identifies  certain  attributes  that  require 
improvement  to  enable  the  device  to  be  capable  of  being  used  to  take  T&R  event  credit 
(e.g.,  visual  or  audio  cues).  The  ability  to  identify  areas  of  possible  improvement  make  it 
possible  to  more  effectively  train  aviation  personnel  and  provide  a  less  costly  alternative 
to  taking  T&R  event  credit  in  an  aircraft  versus  using  an  aviation  training  device.  This 
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potential  savings  could  provide  a  favorable  return  on  investment  for  training  (Owen  and 
Meyers  2012).  The  cost  savings  by  using  an  aviation  training  system  device  instead  of  an 
aircraft  could  easily  be  developed  with  cost  data.  This  thesis  does  not  quantify  the 
potential  cost  savings  but  recommends  further  research  on  the  cost  for  taking  a  T&R 
event  credit  in  an  aircraft  versus  an  aviation  training  system  device. 

The  final  product  of  TDCAP  is  a  results  report.  The  report  summarizes  the 
aviation  training  device’s  capability  to  meet  the  necessary  learning  objectives,  T&R 
events,  and  training  tasks  associated  with  the  training  goal.  Ultimately,  the  report  could 
be  available  to  the  decision-makers  to  determine  if  the  aviation  training  device  should  be 
added  to  the  list  of  approved  aviation  training  devices  in  Appendix  K  of  OPNAVINST 
3710.7U.  The  following  description  is  provided  by  Owen  and  Meyers  for  the  TDCAP 
report: 


This  results  report  represents  the  TDCAP  testing  evidence  and  is  meant  to 
inform  the  user  community’s  accreditation  decisions,  and  should  not  be 
taken  to  imply  that  any  user  community  shall  follow  the  recommendations 
of  the  report.  The  triggers  that  can  initiate  the  TDCAP  are  defined  as 
followed: 


1.  Initial  validation  of  a  training  system.  Initial  delivery  of  a  training 
device  will  require  TDCAP  to  baseline  the  device’s  ability  to  produce 
authentic  trainee  task  performance  to  meet  the  training  task/T&R 
identified  for  the  platform’s  configuration. 


2.  Changes  to  training  system  configuration.  Changes  to  the  hardware  or 
software  configuration  of  the  training  device  may  require  re-certification 
of  associated  training  capabilities. 


3.  Changes  to  platform’s  configuration.  Changes  to  the  hardware  or 
software  configuration  of  the  operational  system  that  has  been  modeled 
may  require  re-certification  of  the  training  device  to  ensure  new 
capabilities  and  events  can  be  taught  with  the  device. 


4.  Task/Mission  update.  The  TDCAP  process  validates  the  training 
device’s  ability  to  support  training  for  changes  to  platform  tasks/mission 
sets. 
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5.  Life  cycle  periodic  re-certification.  Since  the  training  device’s 
performance  may  degrade  over  time,  the  PM  or  user  community  has  the 
authority  to  establish  a  periodic  re-certification  plan  to  ensure  the  device’s 
continued  compliance  with  training  requirements. 


6.  As  requested.  Other  circumstances  may  require  additional  TDCAP 
activities  and  a  TDCAP  analysis  may  be  conducted  at  the  request  of  an 
appropriate  stakeholder  or  stakeholders.  (Owen  and  Meyers  2012,  5) 

The  scenario  provided  above  is  for  an  existing  or  final  phase  in  T&E  for  new 
aviation  training  devices  similar  to  the  START  process.  TDCAP  can  be  applied  during 
the  acquisition  process  unlike  the  START  process.  “The  acquisition  support  portion  of 
TDCAP  is  divided  into  four  primary  sections:  1)  Planning/Preparation;  2)  Requirements 
Generation  (Steps  through  SRR  I);  3)  System  Design  and  Development  (Steps  to  CDR); 
and  4)  Training  Device  T&E”  (Owen  and  Meyers  2012).  Figure  10  and  Figure  11  show 
the  iterative  process  within  the  SETR  process  leading  up  to  SRR-I,  and  after  SRR-I,  to 
delivery. 
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Figure  10.  TDCAP  Pre  SRR-I.  (From  Owen  and  Meyers  2012,  6) 
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Figure  11.  TDCAP  Post  SRR-I.  (From  Owen  and  Meyers  2012,  6) 
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7.  Summary 

There  is  an  overwhelming  amount  of  information  available  about  aviation  training 
systems  management,  acquisition,  design,  engineering,  and  testing.  The  USN  has  a  lot  of 
instructions  and  guidance  currently  available  to  create  a  certification  or  qualification  plan 
for  new  or  existing  aviation  training  system  devices.  The  TDCAP  and  START  processes 
are  achieving  the  same  goal  by  analyzing  existing  aviation  training  system  devices  to 
determine  what  learning  objectives  or  T&R  events  can  be  met.  The  two  processes  also 
quantify  the  capability  of  the  aviation  training  device  to  meet  the  learning  objectives  or 
T&R  events.  However,  START  and  TDCAP  follow  many  of  the  same  methods  described 
in  DoDI  5000.02,  SECNAVINST  5000.2E,  NAVAIRINST  4355. 19D,  and 
NAVAIRINST  3960. 2D.  TDCAP  does  not  show  the  trace  back  to  the  original  DoDI 
5000.02  instruction  to  ensure  the  training  device  team  understands  the  reasoning  behind 
the  process.  TDCAP  is  structured  and  organized  similar  to  a  functional  decomposition  for 
a  system.  In  this  particular  process,  the  goal  is  to  teach  “learning  objectives,”  the 
functions  are  the  different  “tasks,”  and  the  forms  are  the  “attributes.”  As  long  as  the  USN 
training  system  team  can  understand  this  relationship,  they  can  discuss  the  training 
system  device  design  solution  with  stakeholders,  SETR  chairs,  and  program  managers 
who  might  not  have  experience  or  knowledge  with  aviation  training  terminology. 

D.  REVIEW  OF  EXISTING  FAA  AVIATION  TRAINING  SYSTEM  DEVICE 

GUIDANCE 

As  discussed  in  Chapter  III,  the  FAA  regulation  is  referred  to  as  the  “Title  14 
CFR”  and  is  the  overall  requirement  for  qualifying  aviation  training  devices  that  can  be 
used  for  pilot  training.  The  ACs  are  provided  by  the  FAA  for  additional  guidance  to 
comply  with  the  Title  14  CFR  requirements.  This  section  will  analyze  ACs  listed  in  Table 
2. 


1.  Airplane  and  Helicopter  ACs 

The  FAA  developed  a  certification  and  qualification  process  for  the  training 
system  to  match  the  capabilities  of  the  aircraft  it  was  simulating.  “As  technology 
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progressed  and  the  capabilities  of  flight  simulation  were  recognized,  FAR  revisions  were 
made  to  permit  the  increased  use  of  simulators  in  approved  training  programs”  (AC  120- 
40B  1991,  3).  The  ACs  were  created  to  ensure  the  aviation  training  system  met  the 
requirements  for  training.  Simulators  have  been  used  in  the  commercial  aviation  industry 
since  the  1950s.  As  such,  the  FARs  have  been  slowly  revised  over  time  to  ensure  the  new 
aviation  training  devices  meet  the  original  training  system  goals  to  match  the  aircraft  for 
training.  The  most  significant  change  to  the  FAR  was  made  in  the  1970s  as  technology 
improved  to  include  FAR  Amendments  61-62  and  121-108  permitted  additional  use  of 
visual  simulators  in  December  1973.  “Amendments  to  FAR  Section  121.439  permitted 
simulators  approved  for  'the  landing  maneuver'  to  be  substituted  for  the  airplane  in  a  pilot 
recency  of  experience  qualification”  (AC  120-40B  1991,  3).  All  of  these  changes  to  the 
FAR  provided  a  significant  step  towards  the  development  of  Amendments  61-69  and 
121-161  issued  June  24,  1980,  which  contained  the  FAA  Advanced  Simulation  Plan. 
This  trend  showed  an  increased  use  and  demand  for  quality  simulation  by  the  commercial 
aviation  industry. 

The  need  to  create  an  aviation  training  device  certification  and  qualification 
process  was  required  after  the  FAA  allowed  training  system  devices  to  be  used  to  log 
flight  training  time.  Each  FAA  aviation  training  device  AC  is  organized  according  to 
aircraft  type.  All  the  ACs  listed  in  Table  2  follow  the  same  trend.  The  FAA  aviation 
training  device  is  compared  to  the  aircraft  cockpit  depending  on  the  aircraft  type.  In 
addition,  the  FAA  has  different  levels  of  qualification  depending  on  the  level  the  aviation 
training  device  is  constructed  to  represent.  For  example,  an  OFT  has  the  capability  of 
using  motion  and  full  visual  displays  in  the  training  device.  This  aviation  training  device 
would  be  certified  to  the  highest  level  of  qualification  of  representing  the  aircraft  during 
training  scenarios.  The  FAA  does  not  believe  the  highest  level  is  required  for  all  aviation 
training  devices.  The  FAA  Advanced  Simulator  Plan  describes  the  different  phases  as 
Phase  I,  Phase  II,  and  Phase  III.  The  training  credits  for  nonvisual  training  devices  used 
to  be  delineated  in  FAR  Part  61,  Appendix  A,  and  FAR  Part  121,  Appendices  E  and  F  in 
the  past.  Credits  for  Phase  I,  Phase  II,  and  Phase  III  were  contained  in  the  Advanced 
Simulator  Plan.  The  four  different  levels  of  simulation  were  Basic  (nonvisual  and  visual 
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simulators),  Phase  I,  Phase  II,  and  Phase  III.  “Each  of  the  four  levels  is  progressively 
more  complex  than  the  preceding  level  and  each  contains  all  the  features  of  preceding 
levels  plus  the  requirements  for  the  designated  level”  (AC  120-40B  1991,  3).  FAA 
training  system  or  simulator  qualification  guidance  advanced  as  simulator  capabilities 
advanced.  The  FAA  continuously  reviews  the  existing  qualification  criteria  for  simulators 
with  both  government  and  industry  resources.  This  ongoing  effort  requires  active 
participation  by  all  the  stakeholders. 

The  FAA  and  industry  reviewed  the  wide  spectrum  of  aviation  training  system 
devices  or  simulators  and  documented  the  required  standards  and  permitted  uses  for 
training  credits.  As  a  result,  the  FAA  created  a  new  method  to  classify  the  different  levels 
of  simulation  used  by  aviation  training  system  devices.  The  phase  concept  is  not  used  any 
longer.  The  old  phases  were  derived  from  the  FAR  provision  which  allowed  operators  to 
upgrade  their  simulator  in  phases  while  using  the  enhanced  capability  for  training.  This 
method  made  it  possible  for  the  training  system  device  to  be  qualified  at  a  lower  phase 

then  be  used  operationally  for  training  with  the  higher  phase  expectations.  This  old 

method  of  qualification  would  not  discover  and  higher  phase  requirements  since  it  was 
never  followed  to  begin  with.  The  new  process  requires  the  user  to  determine  the  required 
level  of  certification  required  for  training.  The  simulator  is  then  designed,  built,  and 
qualified  to  the  required  level  of  FAA  simulation.  To  meet  the  new  process,  the  FAA 
developed  the  different  qualification  levels  that  exist  today.  The  phases  were  renamed  as 
levels  according  to  the  simulator  capability.  The  new  designations  are  defined  in  FAR 
Part  121  Appendix  H,  and  AC  120-40B.  The  different  FAA  qualification  levels  are: 

•  Fevel  A  -  Visual 

•  Fevel  B  -  Phase  I 

•  Fevel  C  -  Phase  II 

•  Fevel  D  -  Phase  III 

The  FAA  uses  an  Approved  Test  Guide  (ATG)  to  qualify  the  simulator  to  the 
different  Fevel  of  A  through  D.  The  ATG  is  a  document  developed  by  the  FAA  as  a 
means  to  compare  an  aviation  training  device  to  the  aircraft  in  terms  of  performance.  The 
ATG  requires  both  aircraft  and  training  device  data  to  support  the  validation.  The  Master 
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Approval  Test  Guide  (MATG)  is  the  FAA  approved  ATG  that  requires  the  FAA  to 
witness  the  test  results  during  the  qualification  phase  of  testing.  The  MATG  is  used  to 
document  the  results  and  configuration  for  future  modifications  or  enhancements  (AC 
120-40B  1991). 

The  National  Simulator  Evaluation  Program  created  a  qualification  guideline  that 
is  included  in  AC  120-40B  Appendix  1,  Simulator  Standards.  The  FAA  National 
Simulator  Program  Manager  (NSPM)  can  provide  a  recommendation  to  the  Principle 
Operator  Inspector  (POI)  or  certificate  holding  office  to  approve  the  aviation  training 
device  for  use  within  the  training  program  (AC  120-40B  1991). 

Table  4  shows  one  of  the  example  evaluation  criteria  included  in  Appendix  1  of 
AC  120-40B.  There  are  several  different  evaluation  criteria  listed  in  Appendix  1 
arranged  according  to  the  different  system  in  the  aviation  training  device,  such  as  cockpit, 
motion  system,  etc.,  required. 


Table  4.  Example  Airplane  Simulator  Qualification  Criteria  (After  AC  120-40B 

1991,  Al-1) 


Simulator  Level 


2.  General 

A 

B 

C 

D 

Comments 

a)  Cockpit,  a  full-scale  replica  of  the 
airplane  simulated.  Direction  of  movement  of 
controls  and  switches  identical  to  that  in  the 
airplane.  The  cockpit,  for  simulator  purposes, 
consists  of  all  that  space  forward  of  a  cross- 
section  of  the  fuselage  at  the  most  extreme 
aft  setting  of  the  pilots'  seats.  Additional 
required  crewmember  duty  stations  and 
those  required  bulkheads  aft  of  the  pilot 
seats  are  also  considered  part  of  the  cockpit 
and  must  replicate  the  airplane. 

X 

X 

X 

X 

Appendix  2  of  AC  120-40B  covers  the  validation  test  process.  The  validation  test 
process  described  in  Appendix  2  requires  the  test  conductor  and  NSPM  to  record  each 
test  event  in  the  aviation  training  device  and  compare  the  test  data  to  aircraft  source  data. 
During  this  phase  of  testing,  the  FAA  requires  the  entire  aviation  training  device  system 
to  be  tested.  The  FAA  will  not  accept  the  sub-level  component  test  results  for  data.  The 
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system  level  test  is  required  according  to  FAR  part  121,  Appendix  H,  for  Level  D 
qualification.  The  ATG  for  validation  testing  must  include  all  the  details  of  the  test,  such 
as  instrumentation  tolerances,  instrument  calibrations,  and  aircraft  data  tolerances. 
Appendix  2  contains  the  Table  of  Validation  Tests  that  is  required  to  be  completed.  Table 
5  shows  an  example  validation  test  point  to  generate  qualification  data.  Similar  to 
Appendix  1,  Qualification  Criteria,  Appendix  2  contains  several  pages  of  validation  test 
scenarios  (AC  120-40B  1991). 


Table  5.  Example  Airplane  Validation  Test  (After  AC  120-40B  1991,  A2-3) 

I  =  Initial  Evaluation 
R  =  Recurrent  Evaluation 


Test  Tolerance  FI ight  Condition  ication  Requii 


1.  Performance 

A 

B 

C 

D 

Comments 

b)  TAKEOFF 

X 

X 

X 

X 

Ground  Acceleration 

Time  and  Distance 

+/-  5% Time  and  Distance 
or  +/-  5%  Ti  me  a  nd  +/-  200 
Feet  (61  Meters)  of 
Distance 

Ground/Takeoff 

IR 

IR 

IR 

IR 

Unfactored  aircraft  certification 
data  may  be  used.  Acceleration 
Timeand  Distance  should  be 

recorded  for  a  minimum  of  80% 
of  total  segment. 

AC  120-40B  Appendix  3  covers  the  functions  and  subjective  test  phases  for  the 
FAA  qualification.  This  process  allows  aircraft  pilots  to  evaluate  the  simulator  and 
compare  it  to  the  aircraft.  Unlike  the  previous  processes,  this  allows  the  pilot  to  comment 
on  the  handling,  performance,  and  training  capability  without  using  aircraft/training 
device  performance  data.  As  the  title  suggests,  this  process  allows  subjective  comments 
and  might  cover  design  flaws  that  were  missed  during  the  detailed  systematic  testing 
covered  by  Appendix  1  and  2.  Appendix  3  has  a  table  that  is  several  pages  long  similar  to 
Appendix  1  and  2  but  the  pilot  enters  information  in  the  comments  section  about 
handling,  appearance,  etc.  The  table  in  Appendix  3  needs  to  be  completed  during  the 
function  and  subjective  evaluation.  After  completing  all  the  testing  from  Appendices  1,  2, 
and  3,  the  application  letter  in  Appendix  4  must  be  completed  to  request  certification  for 
the  specific  level  the  simulator  was  designed  for,  Level  A,  B,  C,  or  D.  The  application 
letter  quotes  the  specific  requirements  listed  in  the  Title  14  CFR.  Appendix  5  is  the  last 
appendix  in  AC  120-40B.  It  applies  to  aircraft  training  devices  that  will  be  used  to  train 
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pilots  for  windshear.  This  appendix  is  required  for  the  training  devices  that  are  required 
to  comply  with  FAR  Part  121  (AC  120-40B  1991). 

2.  Summary 

FAA  Helicopter  Simulator  Qualification ,  Aircraft  Circular  120-63  (AC  120-63) 
is  formatted  in  exactly  the  same  way  as  AC  120-40B.  The  only  difference  is  that  AC 
120-63  is  tailored  to  helicopters  or  rotary  wing  aviation  training  devices.  AC  120-63  has 
five  appendixes  like  AC  120-40B  but  has  helicopter  features  (AC  120-63  1994). 

After  a  thorough  literature  analysis,  it  is  safe  to  conclude  that  the  FAA  ACs  only 
cover  aviation  training  device  qualification  or  certification.  The  FAA  does  not  appear  to 
cover  what  training  device  configuration  is  the  most  cost  effective  or  the  best 
configuration  to  use  for  training.  Instead,  the  FAA  has  a  very  thorough  process  to  certify 
or  qualify  the  aviation  training  device  is  exactly  like  the  aircraft  it  is  used  to  simulate. 
This  approach  assumes  the  upfront  requirements  generation  within  systems  engineering 
was  performed  to  ensure  the  method  of  training  is  the  proper  method  to  meet  the 
stakeholder’s  requirements.  The  conservative  approach  is  to  use  the  highest  level  of 
certification;  Level  D  for  example,  is  the  closest  configuration  with  tactile  cues  to 
replicate  aircraft  operation.  This  approach  could  lead  to  a  costly  acquisition  when  perhaps 
the  FTD  would  have  been  sufficient.  The  approach  provided  in  the  ACs  is  an  excellent 
approach  for  certification  or  qualification  when  the  proper  aviation  training  functions  that 
lead  to  forms  are  determined.  This  will  determine  the  proper  type  of  aviation  training 
device,  FTD,  OFT,  etc. 

E.  REVIEW  OF  EXISTING  USAF  AVIATION  TRAINING  SYSTEM 

DEVICE  GUIDANCE 

The  USAF  aviation  training  system  guidance  is  similar  to  the  DON’s.  All  the 
USAF  aviation  training  systems  follow  DoDI  5000.02  and  USAF  acquisition  system 
guidance  for  aviation  training  systems.  All  the  AFIs  and  AFPDs  are  traced  back  to  the 
requirements  listed  in  DoDI  5000.02.  AFPD  36-26  is  the  first  layer  in  USAF  policy  for 
acquisition  systems  that  includes  aviation  training  system  devices. 
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1.  Acquisition  Policy,  Guidance,  and  Instructions 

AFPD  36-26  requires  all  USAF  acquisitions  to  follow  the  goals  for  acquisition. 
The  following  statement  is  from  AFPD  36-26  describing  how  the  USAF  is  to  manage 
training  systems  that  includes  aviation: 

Establish  a  learning  capability  that  is  agile  and  robust  enough  to  satisfy 
mission  generated  training  and  mission  rehearsal  requirements  across 
Services,  joint,  interagency,  intergovernmental,  and  multinational 
operations.  Training  must  be  capabilities-based  and  dynamic  in 
responding  to  the  changing  strategic  environment  as  well  as  to 
opportunities  and  challenges  posed  by  technological  transformation. 

(AFPD  36-26  2011,  3) 

This  is  the  AFPD  that  all  of  the  AFIs  are  required  to  comply  with  for  all  of  the 
USAF  acquisitions  including  aviation  training  systems.  AFPD  does  not  provide  training 
system  specific  guidance,  but  does  require  training  systems  to  be  capabilities  based.  This 
instruction  is  the  AFPD  that  links  the  AFIs  to  the  DoDI  5000.02  for  systems  acquisition. 

2.  Training  System  Management  Instruction 

As  mentioned  in  Chapter  III,  the  USAF  aviation  training  systems  are  managed  by 
AFI  36-2251  and  AFI  36-2248.  AFI  36-2251  provides  the  management  framework  for 
acquisition,  programming,  documenting  requirements,  and  testing  in  accordance  with 
DoDI  5000.02  and  AFPD  36-26.  The  requirements  for  aviation  training  system  devices 
are  generated  using  DoDI  5000.02  and  USAF  10-601.  This  process  is  similar  to  any 
other  USAF  acquisition  process.  In  addition,  AFI  36-2251  requires  the  acquisition  team 
to  work  with  the  Air  Force  Career  Field  Managers  (AFCFMs),  Air  Education  and 
Training  Command  (AETC),  and  Training  Pipeline  Managers  (TPMs)  through  a  formal 
Utilization  and  Training  Workshop  (U&TW)  as  detailed  in  AFI  36-2201.  All  the  aviation 
training  devices  should  be  developed  when  supported  by  the  Instructional  System 
Development  (ISD)  analysis.  Heavy  emphasis  is  given  to  ensure  the  ISD  analysis 
includes  safety,  quality  of  training,  and  state  of  readiness.  AFI  36-2251  also  covers  other 
types  of  training  devices,  but  this  thesis  will  only  focus  on  aviation  training  devices  (AFI 
36-2251  2009). 
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AFI  36-225 1  continues  to  describe  the  agencies  responsible  for  training  systems, 
including  the  Secretary  of  the  Air  Force,  Lead  Command  User  Commands  (LC/UCs), 
AETC,  Air  Force  Material  Command  (AFMC),  Air  Force  Space  Command  (AFSPC), 
and  Space  Training  Acquisition  Office  (STAO).  Aviation  training  system  device 
management  roles  and  responsibilities  are  covered  in  Chapter  3  of  AFI  36-2251.  Chapter 
3  describes  how  all  the  different  organizations  will  work  together  to  ensure  the  aviation 
training  system  device  acquisition  or  modification  includes  all  of  the  USAF  stakeholders. 
The  process  is  required  to  ensure  the  training  system  team  is  translating  operational  and 
training  requirements  into  contractual  terms  and  system/technical  performance 
requirements.  This  process  ensures  the  USAF  stakeholders  agree  during  the  requirements 
generation  process  to  ensure  the  correct  capability/training  requirement  is  correctly 
documented.  This  avoids  an  interpretation  of  the  aviation  training  device  requirements 
during  the  contract  or  performance  specification  creation.  The  management  team  works 
at  the  appropriate  level  within  USAF  or  DoD  to  ensure  the  requirements  are  proper  for 
the  level  of  effort.  Chapters  4  and  5  continue  to  describe  the  acquisition  and  management 
structure,  required  meetings,  and  oversight  for  all  training  systems  (AFI  36-2251  2009). 

Aviation  training  system  device  qualification  or  certification,  or  as  the  USAF 
refers  to  as  SIMCERT,  is  covered  in  Chapter  6.  SIMCERT  is  described  as: 

Simulator  certification  (SIMCERT)  ensures  that  Air  Force  prime  mission 
system  simulators/services  and  their  components  support  accurate  and 
credible  training  for  allocated  tasks,  missions,  and  events  including  DMO 
activity,  through  verification  and  validation  of  training  system  hardware 
and  software  performance.  (AFI  36-2251  2009,  13) 

SIMCERT  is  used  in  conjunction  with  Simulator  Validation  (SIMVAL). 
SIMVAL  verifies  and  validates  the  simulated  environment  used  in  the  aviation  training 
system  device  is  accurately  represented.  Both  SIMCERT  and  SIMVAL  compare  the 
aviation  training  system  to  the  prime  mission  that  the  system  is  modeling.  These 
processes  are  used  for  single  aviation  training  devices  and  aviation  training  devices  that 
are  joined  together  for  distributed  network  training.  More  importantly,  SIMCERT  and 
SIMVAL  provide  the  stakeholders  an  assessment  of  the  capabilities  and  limitations  of  the 
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aviation  training  system.  The  results  of  SIMCERT  and  SIMVAL  provide  an  audit  trail  for 
training  effectiveness  and  quality  assurance. 

It  is  up  to  the  Lead  Commands  (LCs)  LCs  to  determine  the  frequency  for 
SIMCERT  or  SIMVAL  evaluations  after  the  initial  certification  or  validation.  The 
evaluation  frequency  is  required  to  be  documented  in  the  System  Training  Plan  (STP).  In 
addition  to  this,  the  SIMCERT  plan  is  required  to  be  documented  in  the  Master 
SIMCERT  Plan  in  accordance  with  Air  Force  Instruction  10-1001  (AFI  10-1001), 
Verification,  Validation  and  Accreditation  (VV&A).  All  new  aviation  training  devices  are 
required  to  complete  the  SIMCERT  process  within  the  first  120  days  upon  delivery  of  a 
new  training  system.  The  SIMCERT  report  supports  the  accreditation  of  the  aviation 
training  system  to  meet  the  training  goals.  SIMVAL  is  used  more  for  mission  training  to 
ensure  the  scenario  is  closely  represented  to  the  scenario  using  the  SIMVAL  report  to 
document  the  results.  AFI  36-2251  does  not  provide  the  framework  for  a  SIMCERT  or 
SIMVAL  plan.  This  instruction  doesn’t  provide  guidance  to  construct  a  SIMCERT  or 
SIMVAL  report  either.  The  instruction  only  requires  USAF  aviation  training  devices  to 
develop,  execute,  and  report  the  SIMCERT  and  SIMVAL  process  (AFI  36-2251  2009). 

3.  Aircrew  Training  System  Management 

AFI  36-2248  is  the  other  USAF  instruction  that  covers  SIMCERT  and  SIMVAL 
for  aviation  training  systems.  This  instruction  covers  operating  and  managing  aircrew 
training  devices  for  Combat  Air  Forces  (CAF).  The  instruction  does  not  provide  any 
additional  guidance,  only  mirrors  the  same  SIMCERT  guidance  provided  in  AFI  36  2251. 
AFI  36-2248  does  not  provide  any  requirement  or  guidance  for  SIMVAL  (AFI  36-2248 
1998). 


4.  Operational  Capability,  Life  Cycle,  Training,  and  Test  Management 
Instructions 

AFI  36-2251  referred  to  several  other  AFIs  to  support  aviation  training  system 
device  management.  These  instructions  are  AFI  10-601,  AFI  63-101,  AFI  36-2201,  and 
AFI  99-103.  AFI  10-601  covers  instructions  to  develop  operational  capability 
requirements  for  all  USAF  systems.  This  instruction  does  not  refer  to  any  training  system 
device  requirements.  It  only  provides  additional  guidance  for  USAF  acquisition  programs 
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to  adhere  to  the  requirements  in  DoDI  5000.02.  AFI  63-101  contains  the  directive  for  the 
overarching  processes  and  procedures  required  for  execution  of  any  USAF  program.  This 
AFI  covers  high  level  acquisition,  engineering,  logistics,  and  disposal  or  all  USAF 
systems.  It  does  not  cover  any  specific  guidance  for  aviation  training  systems  other  than 
the  typical  life  cycle  responsibilities.  AFI  36-2201  provides  instructions  for  overall 
USAF  training,  classroom,  recruitment,  etc..  AFI  36-2201  provides  more  information 
about  the  ISD  process  that  was  referred  to  in  AFI  36-2251.  The  following  description  of 
ISD  was  taken  from  AFI  36-2201: 

The  ISD  process  provides  a  systematic  approach  to  planning,  developing, 
and  implementing  training  and  education.  The  goal  of  ISD  is  to  increase 
the  effectiveness  and  cost  efficiency  of  training  by:  developing  instruction 
based  on  job  performance  requirements;  eliminating  irrelevant  skills  and 
knowledge  instruction  from  courses;  and  ensuring  graduates  acquire  the 
necessary  knowledge,  skills,  and  abilities  to  do  the  job.  (AFI  36-2201 
2010,  8) 

The  ISD  process  looks  at  the  overall  training  process  to  determine  if  it  is  the  right 
training  and  is  effectively  training  USAF  personnel  as  it  was  designed.  The  goals  of  the 
ISD  process  are  similar  to  SIMCERT.  The  ISD  process  is  applied  to  all  training  but 
SIMCERT  is  only  applied  to  training  system  devices.  The  last  AFI,  AFI  99-103,  covers 
capabilities  T&E  for  all  USAF  systems.  AFI  99-103  describes  the  planning,  conduct,  and 
reporting  for  T&E  programs  to  ensure  the  testing  is  conducted  in  an  efficient  manner  by 
combining  test  events  when  possible.  This  instruction  does  not  refer  to  training  system 
devices.  The  methods  used  for  T&E  must  be  applied  to  large  training  systems  that  fall 
within  a  certain  acquisition  category  but  do  not  cover  any  details  about  certification  or 
qualification.  This  instruction  covers  the  entire  verification  side  of  the  systems 
engineering  “V”  model.  The  methods  of  test  described  in  this  instruction  can  be  used 
during  SIMCERT  and  SIMVAL. 

5.  Summary 

The  USAF  has  several  instructions  relating  to  aviation  training  system  devices. 
The  USAF  has  a  well-documented  instruction  that  describes  the  roles  and  responsibilities 
of  every  organization  that  compromises  the  management  team.  The  AFIs  provide 
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excellent  guidance  for  training  system  managers  to  develop  training  system  requirements 
and  ensure  the  proper  stakeholders  are  included.  The  AFIs  list  the  required  organizations 
who  must  participate  in  the  training  system  device  requirements  generation  phase.  AFI 
36-2251  provides  instructions  on  how  to  document  the  management  process,  roles  and 
responsibilities  of  the  different  USAF  organizations,  frequency  of  the  required 
stakeholder  meetings,  the  requirement  to  develop  a  SIMCERT  and  SIMVAL  program, 
and  the  requirement  to  create  a  SIMCERT  report. 

Out  of  all  the  USAF  AFIs,  AFI  36-2251  provided  the  most  information  about 
certification  or  qualification  for  aviation  training  devices.  This  instruction  provides  a 
detailed  plan  for  management  but  lacks  specific  details  about  the  SIMCERT  process.  The 
instruction  only  mentions  that  all  aviation  training  devices  require  a  SIMCERT  but  does 
not  provide  the  framework  to  prepare  a  SIMCERT  plan,  process  to  execute  the 
SIMCERT  plan,  or  the  reporting  content  requirements  after  the  completion  of  SIMCERT. 
The  same  is  true  for  SIMVAL.  Very  little  guidance  was  provided  about  the  SIMVAL 
planning,  execution,  and  reporting. 

F.  REVIEW  OF  EXISTING  USA  AVIATION  TRAINING  SYSTEM  DEVICE 

GUIDANCE 

1.  Acquisition  ARs 

Similar  to  DON  and  USAF,  the  USA  is  required  to  follow  the  DoDI  5000.02 
acquisition  instructions.  AR  350-38  is  the  USA  overarching  AR  for  managing  TADSS. 
This  AR  provides  the  roles  and  responsibilities  for  the  different  team  members  within  the 
Integrated  Product  Team  (IPT)  similar  the  USAF’s  AFI  36-2251.  AR  350-38  describes 
TADSS  as  “System  TADSS  are  designed  and  intended  to  train  individual  and/or 
collective  tasks  associated  with  a  specific  system,  family  of  systems,  or  system  of 
systems  (SoS),  for  example,  UH-60  Helicopters”  (AR  350-38  2013,  1).  This  definition 
for  training  devices  is  the  same  as  DON  and  USAF.  The  regulation  describes  the 
acquisition  requirements  for  new  TADSS.  The  same  acquisition  requirements  are 
required  in  DoDI  500.02.  AR  350-38  describes  the  purpose  of  training  devices.  It 
requires  the  acquisition  team  to  improve  and  sustain  USA  readiness  by  providing  state  of 
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the  art  training  to  enhance  training  realism  and  to  use  very  little  or  no  munitions  during 
training  to  reduce  cost  and  required  land  for  shooting  ranges. 

T&E  is  covered  in  Chapter  5-3,  but  does  not  describe  any  requirements  for 
training  device  certification  or  qualification.  The  T&E  scope  is  similar  to  the  acquisition 
T&E  requirements  listed  in  DoDI  5000.02.  The  training  device  requires  a  Test  and 
Evaluation  Master  Plan  (TEMP)  for  large  acquisition  programs.  The  training  device  T&E 
results  must  be  reported  in  accordance  with  Army  Regulation  73-1  (AR  73-1)  Test  and 
Evaluation  Policy.  The  required  T&E  strategy,  execution,  and  reporting  is  required  to 
comply  with  AR  73-1  and  DoDI  5000.02.  AR  350-38  does  not  require  special 
certification  or  qualification  results  for  training  system  devices.  AR  73-1  does  not 
mention  training  system  devices  within  the  AR.  The  entire  document  describes  the  test 
planning,  execution,  and  reporting  requirements  process  to  comply  with  DoDI  5000.02. 
The  AR  did  not  describe  any  T&E  process  for  training  system  devices. 

2.  Model  and  Simulation 

AR  350-38  did  not  list  any  requirements  for  training  system  devices  to  follow 
Army  Regulation  5-11  (AR  5-11),  Management  of  Army  Models  and  Simulations 
(2005),  but  it  was  listed  as  a  requirement  on  Fort  Rucker’s  website.  Fort  Rucker’s 
Directorate  of  Simulation  uses  AR  5-11  as  a  guideline  to  VV&A  TADSS  (U.S.  Army 
2013). 

AR  5-11  provides  guidance  for  Simulation  and  Modeling  for  Acquisition, 
Requirements  and  Training  (SMART)  concept.  The  SMART  concept  aligns  acquisition, 
and  training  communities  through  the  use  of  M&S.  This  concept  aligns  the  system 
acquisition  with  training  early  in  the  acquisition  phase.  It  requires  acquisition  IPTs  to 
consider  M&S  and  training  when  developing  acquisition  strategies  (AR  5-11  2005). 

This  regulation  provides  guidance  on  how  to  manage  M&S  for  USA  systems  that 
include  training  systems.  AR  5-11  does  not  specifically  describe  a  special  process  for 
TADSS.  The  USA  Directorate  of  Simulation  serves  as  the  accreditation  agent  for  USA 
aviation  training.  Accreditation  is  part  of  the  VV&A  process  and  is  covered  in  Chapter  5. 
AR  5-11  defines  accreditation  as  an  “official  determination  that  the  M&S  is  acceptable 
for  its  intended  purpose”  (AR  5-11  2005,  9).  The  VV&A  process  requires  the  test 
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management  team  to  prepare  an  accreditation  plan  and  report.  The  accreditation  plan  will 
identify  the  accreditation  team;  resources;  milestones;  required  documentation; 
acceptable  criteria;  proposed  accreditation  methodology;  Verification,  Validation,  and 
Certification  (VV&C)  approach.  The  report  will  include  background  information, 
description  of  M&S,  VV&C  results,  analysis  of  V&V  results  to  support  accreditation, 
and  limitations  as  they  affect  the  M&S  project’s  intended  use.  Chapter  5  did  not  list  any 
specific  VV&A  or  VV&C  for  aviation  training  devices  (AR  5-11  2005). 

3.  Army  Training  Guidance 

The  last  AR  related  to  training  devices  is  covered  by  Army  Regulation  350-1  (AR 
350-1),  Army  Training  and  Leader  Development  (2010).  This  AR  prescribes  the  policies, 
procedures,  and  responsibilities  for  developing,  managing,  and  conducting  USA  training 
and  leadership  development.  AR  350-1  lists  all  the  different  USA  organizations  who  are 
responsible  for  utilizing,  managing,  designing,  and  maintaining  USA  training  material 
and  devices  This  AR  does  not  mention  any  process  or  requirement  for  TADSS 
certification  or  qualification. 

4.  Summary 

USA  ARs  discuss  the  importance  of  training  system  devices  for  aviation  training 
but  do  not  provide  a  unique  process  for  certification  or  qualification.  The  USA  uses  the 
M&S  VV&A  and  VV&C  process  to  qualify  or  certify  their  aviation  training  devices.  The 
USAF  and  DON  have  an  M&S  instruction  as  well,  but  they  do  not  refer  to  the  M&S 
instruction  for  qualification  or  certification  of  aviation  training  devices.  The  USA  training 
device  certification  and  qualification  is  focused  more  on  the  M&S  aspect  of  the  system  to 
determine  if  it  is  correctly  modeling  the  system.  The  USA  method  is  focusing  on 
certifying  the  aviation  training  device  is  modeling  the  aircraft. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  GENERAL  DISCUSSION 

There  are  several  instructions  and  different  levels  of  guidance  available  on  the 
subject  of  aviation  training  system  device  certification  and  qualification.  The  FAA, 
USAF,  and  DON  have  several  different  processes  to  perform  aviation  training  system 
device  certification  or  qualification.  All  the  certification  or  qualification  methods  cover 
the  core  objectives  discussed  earlier  to  provide  the  user  some  methodical  evidence  that 
the  training  system  device  is  capable  of  meeting  the  learning  objectives.  In  other  words, 
the  goal  of  certification  and  qualification  is  to  ensure  that  an  aviation  training  system 
device,  if  used  as  designed,  will  result  in  improved  performance  of  the  trainee.  However, 
it  is  the  opinion  of  the  author  that  the  DoD  acquisition  community  is  not  following  their 
own  processes  for  training  system  device  certification  or  qualification.  The  training 
system  requirements  are  not  available  prior  to  SRR-I.  The  training  curriculum  is  never 
available  at  SRR-I  and  should  be.  The  system  design  should  follow  the  requirements 
including  the  curriculum. 

Problems  discovered  with  training  systems  today  would  not  exist  if  the  DoD 
training  system  device  acquisition  teams  followed  the  existing  guidance.  An  example  is 
the  problems  discovered  with  the  USN  T-44A  and  Undergraduate  Military  Flight  Officer 
(UMFO)  training  system  devices.  The  requirements  for  T-44A  were  not  consistent  during 
the  design  phase  which  is  critical  to  certification  and  qualification.  A  design  baseline  was 
not  set  by  all  the  key  stakeholders  early  in  the  acquisition  program.  The  UMFO  program 
should  have  provided  the  expectations  of  the  system.  The  UMFO  training  devices  did  not 
match  the  user’s  expectations  when  it  was  first  delivered.  The  UMFO  instructors 
expected  the  training  device  to  operate  like  the  T-45C  but  it  was  not  documented  as  a 
requirement.  The  problems  discovered  with  the  UMFO  training  system  device  could  have 
been  discovered  during  the  certification  or  qualification  phase  prior  to  delivery  to  the 
customer.  The  certification  and  qualification  process  tests  the  training  system  device  to 
the  requirements.  The  stakeholders  are  involved  during  the  requirements  and  test  phases. 
The  certification  and  qualification  process  is  defined  during  the  requirements  generation 
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process.  This  process  enables  the  stakeholders  to  revise  the  requirements  to  include  any 
missed  requirements  when  the  certification  or  qualification  plan  is  drafted.  All  of  these 
issues  could  have  been  avoided  if  the  existing  process  was  followed.  There  are  very  few 
issues  published  about  the  FAA  training  device  qualification  methods  with  respect  to 
training  objectives  because  they  follow  its  processes. 

There  are  issues  noted  with  DoD  training  systems,  based  upon  the  author’s 
experience,  with  training  system  acquisitions.  Unfortunately,  most  of  the  lessons  learned 
have  not  been  published  or  documented  to  educate  other  training  system  device 
acquisition  teams.  There  are  a  few  teams  that  have  followed  the  START  process  to 
improve  the  quality  of  the  training  system  by  performing  post-delivery  modifications. 
However,  as  TDCAP  suggests  these  issues  should  have  been  corrected  during  the  design 
phase. 

Given  today's  news  headlines,  it  is  likely  that  most  training  system  acquisitions 
are  facing  scrutiny  over  budgets  and  acquisition  timelines.  Most  of  the  processes 
identified  in  this  thesis  are  described  as  separate  processes  outside  of  the  typical 
acquisition  SE  process.  Certification  is  part  of  the  SE  process  and  should  be  included  in 
the  acquisition  strategy,  not  as  a  separate  process  that  can  be  easily  removed  when 
schedules  and  budgets  are  challenged.  The  method  of  qualifying  a  training  system  device 
is  not  different  from  qualifying  any  other  product  that  has  the  purpose  to  meet  a  user’s 
need  and  should  be  treated  as  such.  It  is  the  author’s  opinion  that  the  next  DoD  training 
system  acquisition  should  incorporate  the  processes  described  in  this  thesis  for 
certification  or  qualification  and  modify  the  existing  guidance  if  there  are  any  lessons 
learned.  The  acquisition  team  should  not  follow  a  certification  or  qualification  method 
just  because  it  exists.  The  team  should  apply  critical  thinking  to  ensure  the  certification  or 
qualification  process  meets  the  original  requirement.  This  provides  evidence  that  the 
training  device  meets  the  original  learning  objectives  for  T&R.  It  is  the  author’s  opinion 
that  if  the  acquisition  team  can  accomplish  this,  the  certification  or  qualification  process 
cannot  be  separated  during  program  execution  when  budgets  and  schedules  are 
challenged. 


46 


B.  CONCLUSIONS 


1.  Stakeholders,  Requirements,  and  Certification 

This  thesis  answered  the  following  research  questions: 

What  is  the  benefit  to  developing  a  certification  and  qualification  process? 

Can  it  be  quantified? 

Aviation  system  device  certification  and  qualification  is  an  important  phase 
within  the  T&E  systems  “V”  model.  Certification  or  qualification  provides  evidence  to 
the  users  that  the  system  as  designed  meets  the  original  training  goal.  The  training  system 
device  is  a  correct  representation  of  the  aircraft  platform  it  was  designed  for.  The  USN, 
USMC,  USAF,  and  USA,  have  different  approaches  to  certify  or  qualify  aviation  system 
training  devices.  Although  the  techniques  differ,  they  all  meet  their  original  certification 
or  qualification  intent,  to  model  the  aircraft,  or  to  ensure  the  system  is  acceptable  to 
provide  training.  A  successful  aviation  training  system  requires  the  services  to  develop 
good  requirements  to  ensure  the  aviation  training  system  training  goals  are  well 
understood  by  all  the  stakeholders. 

The  DON  uses  a  combination  of  aviation  training  devices  and  aircraft  to  provide 
T&R.  Flying  in  an  aircraft  for  training  is  very  costly.  It  is  more  cost  effective  to  use  a 
training  device  for  training  than  an  aircraft.  As  a  result,  USN  and  USMC  are  interested  in 
looking  at  using  more  aviation  training  system  devices  to  provide  more  T&R  credit  than 
the  aircraft.  The  USN  START  process  maps  all  of  the  learning  objectives  to  T&R  events. 

What  were  the  stakeholder's  requirements  for  the  training  systems?  How  are 

the  requirements  specified,  and  how  are  they  subsequently  used  in  the 

development  process? 

The  USAF  training  system  management  process  is  well  documented.  The  USAF 
process  provides  guidance  to  ensure  all  the  training  system  stakeholders  are  involved 
early  in  the  acquisition  process.  This  method  would  ensure  all  the  stakeholder’s 
requirements  for  the  new  aviation  training  system  are  well  understood.  The  DON,  USN 
and  USMC  describe  an  upfront  FEA  analysis  but  do  not  describe  the  method  to  ensure  all 
the  proper  DON  stakeholders  are  involved  during  the  upfront  requirements  generation 
process  or  through  the  design  phase.  This  process  could  be  overlooked  during  the  team 
formation  or  requirements  generation  phase  for  training  systems.  The  USAF  does  not 
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provide  a  lot  of  detail  for  certification  or  qualification  for  training  devices  but  its  policy 
requires  SIMCERT  to  be  performed.  It  is  the  author's  recommendation  that  the  USN 
certification  process  be  revised  to  include  the  USAF  requirements  generation  process  and 
the  DON  certification  process.  The  USAF  has  a  well-documented  instruction  that 
describes  the  roles  and  responsibilities  of  every  organization  that  compromises  the 
management  team.  The  AFIs  provide  excellent  guidance  for  training  system  managers  to 
develop  training  system  requirements  and  ensure  the  proper  stakeholders  are  included. 
The  USN  and  USAF  certification  process  is  not  sufficient  to  meet  the  USN  certification 
requirements  individually.  The  USN  TDCAP  lacks  requirements  generation  guidance. 

USN  TDCAP  is  applied  during  the  acquisition  process.  TDCAP  is  included  in  the 
planning,  requirements  generation  (steps  to  SRR  I),  system  design  (steps  to  CDR),  and 
T&E.  The  USN  and  USAF  certification  processes  should  be  combined  to  meet  the  USN 
stakeholders'  requirements  for  training  system  certification  or  qualification. 

The  FAA  has  a  well-documented  process  for  training  system  certification  or 
qualification.  However,  the  FAA  does  not  provide  guidance  to  the  developer  about  what 
level  of  “realism”  is  adequate  for  meeting  the  learning  objectives.  The  most  realistic 
aviation  training  system  is  Fevel  D,  however,  that  is  also  the  most  costly  training  system 
to  develop.  A  lower  Fevel  simulator  might  be  adequate  to  accomplish  the  learning 
objectives.  The  FAA  developed  a  certification  and  qualification  process  for  the  training 
system  to  match  the  capabilities  of  the  aircraft  it  was  simulating. 

2.  Certification  Documentation 

This  thesis  researched  several  different  military,  commercial,  and  federal  agencies 
to  develop  a  list  of  existing  certification  and  qualification  processes. 

What  organizations  currently  have  a  training  system  certification  and 

qualification  process?  Are  there  any  well  accepted  "best  practices"  in  the 

industry? 

Why  did  the  organization  develop  a  certification  and  qualification  process? 

There  is  an  overwhelming  amount  of  information  available  about  aviation  training 
systems  management,  acquisition,  design,  engineering,  and  testing.  The  USN  has  a  lot  of 
instructions  and  guidance  currently  available  to  create  a  certification  or  qualification  plan 
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for  new  or  existing  aviation  training  system  devices.  The  USN  TDCAP  and  START 
processes  are  achieving  the  same  goal  by  analyzing  existing  aviation  training  system 
devices  to  determine  what  learning  objectives  or  T&R  events  can  be  met.  The  two 
processes  also  quantify  the  capability  of  the  aviation  training  device  to  meet  the  learning 
objectives  or  T&R  events.  TDCAP  is  structured  and  organized  similar  to  a  functional 
decomposition  for  a  system.  In  this  particular  process,  the  goal  is  to  teach  “learning 
objectives,”  the  functions  are  the  different  “tasks,”  and  the  forms  are  the  “attributes.” 

The  FAA  ACs  were  created  to  ensure  the  aviation  training  system  met  the 
requirements  for  training.  The  need  to  create  an  aviation  training  device  certification  and 
qualification  process  was  required  after  the  FAA  allowed  training  system  devices  to  be 
used  to  log  flight  training  time.  Each  FAA  aviation  training  device  AC  is  organized 
according  to  aircraft  type. 

The  USAF  has  several  instructions  relating  to  aviation  training  system  devices. 
AFI  36-2251  provided  the  most  information  about  certification  or  qualification  for 
aviation  training  devices.  This  instruction  provides  a  detailed  plan  for  management  but 
lacks  specific  details  about  the  SIMCERT  process.  The  instruction  only  mentions  that  all 
aviation  training  devices  require  a  SIMCERT  but  does  not  provide  the  framework  to 
prepare  a  SIMCERT  plan,  process  to  execute  the  SIMCERT  plan,  or  the  reporting 
content  requirements  after  the  completion  of  SIMCERT. 

After  a  considerable  amount  of  literature  research,  it  is  a  safe  assumption  to 
conclude  that  there  is  no  USA  specific  policy  or  guidance  for  training  system  certification 
or  qualification. 

What  is  an  acceptable  reporting  process  for  the  certification  process? 

The  final  product  of  the  USN  TDCAP  is  a  results  report.  The  report  summarizes 
the  aviation  training  device’s  capability  to  meet  the  necessary  learning  objectives,  T&R 
events,  and  training  tasks  associated  with  the  training  goal.  Ultimately,  the  report  could 
be  available  to  the  decision-makers  to  determine  if  the  aviation  training  device  should  be 
added  to  the  list  of  approved  aviation  training  devices. 
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The  USAF  provides  the  stakeholders  a  SIMCERT  report.  The  SIMCERT  report 
provides  the  stakeholders  an  assessment  of  the  capabilities  and  limitations  of  the  aviation 
training  system.  The  results  of  SIMCERT  provide  an  audit  trail  for  training  effectiveness 
and  quality  assurance. 

What  are  the  tradeoffs  for  not  completing  all  of  the  recommended  test  events 
for  certification?  What  is  the  true  "return  on  investment"  made  in 
certification  and  qualification  process? 

The  training  device  would  not  be  certified  for  certain  learning  objectives  if  the 
training  device  is  not  fully  tested.  The  learning  objectives  would  need  to  be  accomplished 
in  the  aircraft.  The  cost  savings  by  using  a  training  system  device  instead  of  an  aircraft 
could  easily  be  developed  with  cost  data.  This  thesis  does  not  quantify  the  potential  cost 
savings  but  recommends  further  research  on  the  cost  for  taking  a  T&R  event  credit  in  an 
aircraft  versus  an  aviation  training  system  device.  This  analysis  requires  additional  time 
and  cost  data. 

C.  RECOMMENDATIONS 

Through  analysis  of  all  the  existing  training  system  device  certification  and 
qualification  processes,  the  following  recommendations  for  USN  certification  or 
qualification  process  are  suggested  to  address  the  weaknesses  listed  in  the  conclusions. 

1.  Provide  guidance  for  training  system  device  requirements  generation. 
Recommend  listing  all  the  necessary  stakeholders’  roles  and  responsibilities  for 
developing  training  system  device  goals,  functions,  and  design  limitations  to 
include  required  equipment.  This  will  ensure  the  proper  stakeholders  are  involved. 

2.  Ensure  the  FEA  analysis  and  the  USN  users  are  involved  through  the  entire 
training  system  device  acquisition  process,  from  requirements  generation  to 
certification  or  qualification.  This  will  help  provide  testable  requirements  during 
the  certification  process. 

3.  Develop  or  use  an  existing  requirements  tractability  tool  to  ensure  the  design 
meets  the  original  learning  objectives  or  required  skills.  This  method  will  help 
during  the  certification  or  qualification  phase  to  show  the  impact  of  any 
deficiencies  discovered  during  T&E. 

4.  Include  a  process  to  include  FEA  authors  during  the  design  and  test  phases. 
TDCAP  describes  the  FEA  traceability  through  the  system  specification  and 
RTVM  but  does  not  describe  the  process  to  involve  the  USN  users  through  the 
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design  process.  This  will  help  define  requirements  and  avoid  potential 
requirements  creep. 

5.  Recommend  USN  develop  a  process  to  add  aviation  training  devices  to 
OPNAVINST  3710.7U.  This  will  avoid  potential  inconsistencies  for  adding 
training  system  devices  to  the  approved  list. 

6.  Include  a  section  in  TDCAP  that  describes  how  the  process  complies  with  the 
DoDI  5000.02  process  but  tailored  to  training  system  devices.  All  the  other 
services  provide  this  link  in  their  instructions  to  show  traceability  back  to  DoDI 
5000.02.  This  will  help  show  tractability  to  the  overarching  DoD  SE  process. 

7.  Include  the  TDCAP  as  part  of  the  acquisition  strategy  for  training  system 
device  acquisition.  Do  not  describe  TDCAP  as  a  separate  process  but  identify 
how  TDCAP  will  meet  the  DoDI  5000.02  and  SECNAVINST  5000.2E 
requirements  for  SE  for  training  system  design,  procurement,  and  test.  This  will 
help  training  system  acquisition  programs  justify  the  certification  process. 

8.  Follow  the  existing  TDCAP  guidance  for  one  project  then  correct  the  process  to 
meet  the  original  intent  of  TDCAP  without  creating  a  new  process  with  a  different 
name  but  same  goals.  The  START  and  TDCAP  process  are  very  similar  but  can 
be  confusing  with  different  names. 

9.  Provide  a  method  to  train  the  NAWCTSD  workforce  about  TDCAP  and  the 
similarities  to  systems  engineering.  This  will  help  the  workforce  understand  the 
relationships  between  the  learning  objectives  and  the  system  design. 

10.  Develop  a  T&E  tool  similar  to  a  requirement  traceability  tool  but  link  any 
deficiencies  discovered  during  certification  or  qualification  testing  back  to  the 
original  learning  objective.  This  will  help  the  program  manager  understand  the 
impact  to  the  learning  objective. 

D.  AREAS  FOR  FURTHER  RESEARCH 

This  thesis  analyzed  published  instructions  and  guidance  about  certification  or 
qualification  for  training  devices.  Areas  of  possible  future  research  would  include: 

1.  Contact  each  service  to  determine  what  agency  is  responsible  for  conducting 
certification  or  qualification  for  training  devices.  Ask  the  agency  if  there  is  any 
existing  guidance  for  conducting  certification  or  qualification  that  is  not  covered 
in  a  published  instruction. 

2.  Conduct  a  USN  case  study  on  a  training  device  system  to  include  requirements 
generation,  design,  test,  and  fielding,  to  determine  areas  of  improvement  for 
TDCAP. 
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3.  Investigate  cost  benefits  from  transferring  training  from  the  aircraft  to  the 
aviation  training  device.  Include  benefits  or  impacts  to  the  learning  objective  by 
using  the  aviation  training  device  instead  of  the  aircraft. 


52 


LIST  OF  REFERENCES 


Chief  of  Naval  Operations.  2009.  Naval  Air  Training  and  Operating  Procedures 

Standardization  General  Flight  and  Operating  Instructions.  (OPNAVINST 
3710.7U).  Washington,  DC:  Department  of  the  Navy. 

Airlines  for  America.  2013.  “Annual  Crude  Oil  and  Jet  Fuel  Prices.”  Accessed  July  20, 
2013.  http://www.airlines.org/Pages/Annual-Crude-Oil-and-Jet-Fuel-Prices.aspx. 

Commandant  of  the  Marine  Corps.  2011.  Aviation  Training  and  Readiness  (T&R) 

Program  Manual.  (NAVMC  3500. 14C).  Washington,  DC:  Department  of  the 
Navy. 

Department  of  Defense.  2009.  Defense  Acquisition  Guidebook.  (DAG).  Washington,  DC: 
Department  of  Defense. 

Department  of  Defense.  2001.  Instructional  Systems  Development/Systems  Approach  to 
Training  and  Education  Part  2  of  5  Parts.  (MIL-HDBK-29612-2A).  Washington, 
DC:  Department  of  Defense. 

Department  of  the  Army.  2013.  Policies  and  Management  for  Training  Aids,  Devices, 
Simulators,  and  Simulations.  (AR  350-38).  Washington,  DC:  Department  of  the 
Army. 

- .  2005.  Management  of  Army  Models  and  Simulations.  (AR  5-11).  Washington, 

DC:  Department  of  the  Army. 

Federal  Aviation  Administration.  1991.  Airplane  Simulator  Qualification.  (AC  120-40B). 
Washington,  DC:  Department  of  Transportation. 

- .  1992.  Airplane  Flight  Training  Device  Qualification.  (AC  120  -  45A). 

Washington,  DC:  Department  of  Transportation. 

- .  1994.  Helicopter  Simulator  Qualification.  (AC  120-63).  Washington,  DC: 

Department  of  Transportation. 

- .  n.d.  Aviation  Technician  Handbook:  Chapter  12,  Publications,  Forms,  and 

Records.  Washington,  DC:  Department  of  Transportation. 

- .  n.d.  “National  Simulator  Program  (NSP).”  Accessed  June  8,  2013. 

http://www.faa.gov/about/initiatives/nsp. 

- .  2013.  “Government  Printing  Office:  Title  14  Code  of  Federal  Regulations.” 

Accessed  June  8,  2013.  http://www.ecfr.gov/cgi-bin/text-idx?c=ecfr&tpl=/ 
ecfrbro  wse/Title  14/1 4tab_02  .tpl . 


53 


INCOSE.  2004.  “International  Council  of  Systems  Engineering.”  Accessed  June  6,  2013. 
http://www.incose.org/practice/whatissystemseng.aspx. 

Jenkinson,  Paul.  1983.  TRADEVMAN  3  &  2:  Naval  Education  and  Traning  Center. 
Washington,  DC:  Government  Printing  Office. 

Maier,  Mark  W.,  and  Eberhardt  Rechtin.  2009.  The  Art  of  Systems  Architecting.  Boca 
Raton:  Taylor  &  Francis. 

Owen,  John,  and  John  Meyers.  2012.  “Training  Device  Certification  and  Accreditation 
Process.”  Paper  presented  at  the  Interservice/Industry  Training,  Simulation,  and 
Education  Conference  (I/ITSEC),  Orlando,  December. 

Pierce,  Clark.  2012.  “Media  Get  Close  Look  at  P-8  Training  System.”  Tax  Air  News. 

September  27.  Accessed  June  25,  2013.  http://jacksonville.com/military/jax-air- 
news/2012-09- 12/story /media-get-close-look-p-8a-training-program. 

Secretary  of  the  Air  Force.  2010.  Air  Force  Training  Program.  (AFI  36-2201). 
Washington,  DC:  Department  of  the  Air  Force. 

- .  1998.  Operation  and  Management  of  Aircrew  Training  Devices.  (AFI  36-2248). 

Washington,  DC:  Department  of  the  Air  Force. 

- .  2009.  Management  of  Air  Force  Training  Systems.  (AFI  36-2251).  Washington, 

DC:  Department  of  the  Air  Force. 

- .  2011.  Total  Force  Development.  (AFPD  36-26).  Washington,  DC:  Department  of 

the  Air  Force. 

U.S.  Army.  2013.  “U.S.  Army  Center  of  Excellence  and  Fort  Rucker.”  Accessed  July  4, 
2013.  http://www-rucker.army.mil/usaace/directorates/dos/index.html. 

Wood,  Janice.  2009.  “King  Air  350  Pro  Line  21  Sim  Approved”  General  Aviation  News. 
January  17.  Accessed  June  25,  2013.  http://www.generalaviationnews.com/2012/ 
0 1  /king-air-3 5 0-pro-line-  2 1  -  sim-  approved/. 


54 


INITIAL  DISTRIBUTION  LIST 


1.  Defense  Technical  Information  Center 
Ft.  Belvoir,  Virginia 

2.  Dudley  Knox  Library 
Naval  Postgraduate  School 
Monterey,  California 


55 


